Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] qp memory allocation problem
From: Shamis, Pavel (shamisp_at_[hidden])
Date: 2011-09-12 12:39:00


Alternative solution for the problem is updating your memory limits
Please see below: http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages

Apparently you memory limit is low and the driver fails to create QPs

What happens when you add the following to your mpirun command?

-mca btl_openib_receive_queues S,4096,128:S,12288,128:S,65536,12

And if you have connectX-2/or 3 device I would suggest replace "S" by "X" - X enables much more efficient XRC QP, instead of SRQ.

OMPI Developers:

Maybe we should consider disabling the use of per-peer queue pairs by default. Do they buy us anything? For what it is worth, we have stopped using them on all of our large systems here at LANL.

It is cons-and-pros of using per-peer queues. It really depends on application behavior.
I would suggest some dynamic-adjustment solution. If NP > some_threshould -> switch to SRQ or XRC.

Also I would recommend to print our some informative error message for "qp errno says Cannot allocate memory" error.

Regards,
Pasha.

Thanks,

Samuel K. Gutierrez
Los Alamos National Laboratory

On Sep 12, 2011, at 9:23 AM, Blosch, Edwin L wrote:

I am getting this error message below and I don’t know what it means or how to fix it. It only happens when I run on a large number of processes, e.g. 960. Things work fine on 480, and I don’t think the application has a bug. Any help is appreciated…

[c1n01][[30697,1],3][connect/btl_openib_connect_oob.c:464:qp_create_one] error creating qp errno says Cannot allocate memory
[c1n01][[30697,1],4][connect/btl_openib_connect_oob.c:464:qp_create_one] error creating qp errno says Cannot allocate memory

Here’s the mpirun command I used:
mpirun --prefix /usr/mpi/intel/openmpi-1.4.3 --machinefile <host file> -np 960 --mca btl ^tcp --mca mpool_base_use_mem_hooks 1 --mca mpi_leave_pinned 1 -x LD_LIBRARY_PATH -x MPI_ENVIRONMENT=1 <application name + arguments>

Here’s the applicable hardware from the /usr/mpi/intel/openmpi-1.4.3/share/openmpi/mca-btl-openib-device-params.ini file:
# A.k.a. ConnectX
[Mellanox Hermon]
vendor_id = 0x2c9,0x5ad,0x66a,0x8f1,0x1708,0x03ba,0x15b3,0x119f
vendor_part_id = 25408,25418,25428,26418,26428,25448,26438,26448,26468,26478,26488
use_eager_rdma = 1
mtu = 2048
max_inline_data = 128

And this is the output of ompi_info –param btl openib:
                MCA btl: parameter "btl_base_verbose" (current value: "0", data source: default value)
                         Verbosity level of the BTL framework
                MCA btl: parameter "btl" (current value: <none>, data source: default value)
                         Default selection set of components for the btl framework (<none> means use all components that can be found)
                MCA btl: parameter "btl_openib_verbose" (current value: "0", data source: default value)
                         Output some verbose OpenIB BTL information (0 = no output, nonzero = output)
                MCA btl: parameter "btl_openib_warn_no_device_params_found" (current value: "1", data source: default value, synonyms:
                         btl_openib_warn_no_hca_params_found)
                         Warn when no device-specific parameters are found in the INI file specified by the btl_openib_device_param_files MCA
                         parameter (0 = do not warn; any other value = warn)
                MCA btl: parameter "btl_openib_warn_no_hca_params_found" (current value: "1", data source: default value, deprecated, synonym
                         of: btl_openib_warn_no_device_params_found)
                         Warn when no device-specific parameters are found in the INI file specified by the btl_openib_device_param_files MCA
                         parameter (0 = do not warn; any other value = warn)
                MCA btl: parameter "btl_openib_warn_default_gid_prefix" (current value: "1", data source: default value)
                         Warn when there is more than one active ports and at least one of them connected to the network with only default GID
                         prefix configured (0 = do not warn; any other value = warn)
                MCA btl: parameter "btl_openib_warn_nonexistent_if" (current value: "1", data source: default value)
                         Warn if non-existent devices and/or ports are specified in the btl_openib_if_[in|ex]clude MCA parameters (0 = do not
                         warn; any other value = warn)
                MCA btl: parameter "btl_openib_want_fork_support" (current value: "-1", data source: default value)
                         Whether fork support is desired or not (negative = try to enable fork support, but continue even if it is not
                         available, 0 = do not enable fork support, positive = try to enable fork support and fail if it is not available)
                MCA btl: parameter "btl_openib_device_param_files" (current value:
                         "/usr/mpi/intel/openmpi-1.4.3/share/openmpi/mca-btl-openib-device-params.ini", data source: default value, synonyms:
                         btl_openib_hca_param_files)
                         Colon-delimited list of INI-style files that contain device vendor/part-specific parameters
                MCA btl: parameter "btl_openib_hca_param_files" (current value:
                         "/usr/mpi/intel/openmpi-1.4.3/share/openmpi/mca-btl-openib-device-params.ini", data source: default value, deprecated,
                         synonym of: btl_openib_device_param_files)
                         Colon-delimited list of INI-style files that contain device vendor/part-specific parameters
                MCA btl: parameter "btl_openib_device_type" (current value: "all", data source: default value)
                         Specify to only use IB or iWARP network adapters (infiniband = only use InfiniBand HCAs; iwarp = only use iWARP NICs;
                         all = use any available adapters)
                MCA btl: parameter "btl_openib_max_btls" (current value: "-1", data source: default value)
                         Maximum number of device ports to use (-1 = use all available, otherwise must be >= 1)
                MCA btl: parameter "btl_openib_free_list_num" (current value: "8", data source: default value)
                         Intial size of free lists (must be >= 1)
                MCA btl: parameter "btl_openib_free_list_max" (current value: "-1", data source: default value)
                         Maximum size of free lists (-1 = infinite, otherwise must be >= 0)
                MCA btl: parameter "btl_openib_free_list_inc" (current value: "32", data source: default value)
                         Increment size of free lists (must be >= 1)
                MCA btl: parameter "btl_openib_mpool" (current value: "rdma", data source: default value)
                         Name of the memory pool to be used (it is unlikely that you will ever want to change this
                MCA btl: parameter "btl_openib_reg_mru_len" (current value: "16", data source: default value)
                         Length of the registration cache most recently used list (must be >= 1)
                MCA btl: parameter "btl_openib_cq_size" (current value: "1000", data source: default value, synonyms: btl_openib_ib_cq_size)
                         Size of the OpenFabrics completion queue (will automatically be set to a minimum of (2 * number_of_peers *
                         btl_openib_rd_num))
                MCA btl: parameter "btl_openib_ib_cq_size" (current value: "1000", data source: default value, deprecated, synonym of:
                         btl_openib_cq_size)
                         Size of the OpenFabrics completion queue (will automatically be set to a minimum of (2 * number_of_peers *
                         btl_openib_rd_num))
                MCA btl: parameter "btl_openib_max_inline_data" (current value: "-1", data source: default value, synonyms:
                         btl_openib_ib_max_inline_data)
                         Maximum size of inline data segment (-1 = run-time probe to discover max value, otherwise must be >= 0). If not
                         explicitly set, use max_inline_data from the INI file containing device-specific parameters
                MCA btl: parameter "btl_openib_ib_max_inline_data" (current value: "-1", data source: default value, deprecated, synonym of:
                         btl_openib_max_inline_data)
                         Maximum size of inline data segment (-1 = run-time probe to discover max value, otherwise must be >= 0). If not
                         explicitly set, use max_inline_data from the INI file containing device-specific parameters
                MCA btl: parameter "btl_openib_pkey" (current value: "0", data source: default value, synonyms: btl_openib_ib_pkey_val)
                         OpenFabrics partition key (pkey) value. Unsigned integer decimal or hex values are allowed (e.g., "3" or "0x3f") and
                         will be masked against the maximum allowable IB paritition key value (0x7fff)
                MCA btl: parameter "btl_openib_ib_pkey_val" (current value: "0", data source: default value, deprecated, synonym of:
                         btl_openib_pkey)
                         OpenFabrics partition key (pkey) value. Unsigned integer decimal or hex values are allowed (e.g., "3" or "0x3f") and
                         will be masked against the maximum allowable IB paritition key value (0x7fff)
                MCA btl: parameter "btl_openib_psn" (current value: "0", data source: default value, synonyms: btl_openib_ib_psn)
                         OpenFabrics packet sequence starting number (must be >= 0)
                MCA btl: parameter "btl_openib_ib_psn" (current value: "0", data source: default value, deprecated, synonym of:
                         btl_openib_psn)
                         OpenFabrics packet sequence starting number (must be >= 0)
                MCA btl: parameter "btl_openib_ib_qp_ous_rd_atom" (current value: "4", data source: default value)
                         InfiniBand outstanding atomic reads (must be >= 0)
                MCA btl: parameter "btl_openib_mtu" (current value: "3", data source: default value, synonyms: btl_openib_ib_mtu)
                         OpenFabrics MTU, in bytes (if not specified in INI files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes,
                         4=2048 bytes, 5=4096 bytes
                MCA btl: parameter "btl_openib_ib_mtu" (current value: "3", data source: default value, deprecated, synonym of:
                         btl_openib_mtu)
                         OpenFabrics MTU, in bytes (if not specified in INI files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes,
                         4=2048 bytes, 5=4096 bytes
                MCA btl: parameter "btl_openib_ib_min_rnr_timer" (current value: "25", data source: default value)
                         InfiniBand minimum "receiver not ready" timer, in seconds (must be >= 0 and <= 31)
                MCA btl: parameter "btl_openib_ib_timeout" (current value: "20", data source: default value)
                         InfiniBand transmit timeout, plugged into formula: 4.096 microseconds * (2^btl_openib_ib_timeout)(must be >= 0 and <=
                         31)
                MCA btl: parameter "btl_openib_ib_retry_count" (current value: "7", data source: default value)
                         InfiniBand transmit retry count (must be >= 0 and <= 7)
                MCA btl: parameter "btl_openib_ib_rnr_retry" (current value: "7", data source: default value)
                         InfiniBand "receiver not ready" retry count; applies *only* to SRQ/XRC queues. PP queues use RNR retry values of 0
                         because Open MPI performs software flow control to guarantee that RNRs never occur (must be >= 0 and <= 7; 7 =
                         "infinite")
                MCA btl: parameter "btl_openib_ib_max_rdma_dst_ops" (current value: "4", data source: default value)
                         InfiniBand maximum pending RDMA destination operations (must be >= 0)
                MCA btl: parameter "btl_openib_ib_service_level" (current value: "0", data source: default value)
                         InfiniBand service level (must be >= 0 and <= 15)
                MCA btl: parameter "btl_openib_use_eager_rdma" (current value: "-1", data source: default value)
                         Use RDMA for eager messages (-1 = use device default, 0 = do not use eager RDMA, 1 = use eager RDMA)
                MCA btl: parameter "btl_openib_eager_rdma_threshold" (current value: "16", data source: default value)
                         Use RDMA for short messages after this number of messages are received from a given peer (must be >= 1)
                MCA btl: parameter "btl_openib_max_eager_rdma" (current value: "16", data source: default value)
                         Maximum number of peers allowed to use RDMA for short messages (RDMA is used for all long messages, except if
                         explicitly disabled, such as with the "dr" pml) (must be >= 0)
                MCA btl: parameter "btl_openib_eager_rdma_num" (current value: "16", data source: default value)
                         Number of RDMA buffers to allocate for small messages(must be >= 1)
                MCA btl: parameter "btl_openib_btls_per_lid" (current value: "1", data source: default value)
                         Number of BTLs to create for each InfiniBand LID (must be >= 1)
                MCA btl: parameter "btl_openib_max_lmc" (current value: "0", data source: default value)
                         Maximum number of LIDs to use for each device port (must be >= 0, where 0 = use all available)
                MCA btl: parameter "btl_openib_enable_apm_over_lmc" (current value: "0", data source: default value)
                         Maximum number of alterative paths for each device port (must be >= -1, where 0 = disable apm, -1 = all availible
                         alternative paths )
                MCA btl: parameter "btl_openib_enable_apm_over_ports" (current value: "0", data source: default value)
                         Enable alterative path migration (APM) over different ports of the same device (must be >= 0, where 0 = disable APM
                         over ports , 1 = enable APM over ports of the same device)
                MCA btl: parameter "btl_openib_use_async_event_thread" (current value: "1", data source: default value)
                         If nonzero, use the thread that will handle InfiniBand asyncihronous events
                MCA btl: parameter "btl_openib_buffer_alignment" (current value: "64", data source: default value)
                         Prefered communication buffer alignment, in bytes (must be > 0 and power of two)
                MCA btl: parameter "btl_openib_use_message_coalescing" (current value: "1", data source: default value)
                         Use message coalescing
                MCA btl: parameter "btl_openib_cq_poll_ratio" (current value: "100", data source: default value)
                         how often poll high priority CQ versus low priority CQ
                MCA btl: parameter "btl_openib_eager_rdma_poll_ratio" (current value: "100", data source: default value)
                         how often poll eager RDMA channel versus CQ
                MCA btl: parameter "btl_openib_hp_cq_poll_per_progress" (current value: "10", data source: default value)
                         max number of completion events to process for each call of BTL progress engine
                MCA btl: information "btl_openib_have_fork_support" (value: "1", data source: default value)
                         Whether the OpenFabrics stack supports applications that invoke the "fork()" system call or not (0 = no, 1 = yes).
                         Note that this value does NOT indicate whether the system being run on supports "fork()" with OpenFabrics applications
                         or not.
                MCA btl: parameter "btl_openib_exclusivity" (current value: "1024", data source: default value)
                         BTL exclusivity (must be >= 0)
                MCA btl: parameter "btl_openib_flags" (current value: "310", data source: default value)
                         BTL bit flags (general flags: SEND=1, PUT=2, GET=4, SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags
                         only used by the "dr" PML (ignored by others): ACK=16, CHECKSUM=32, RDMA_COMPLETION=128)
                MCA btl: parameter "btl_openib_rndv_eager_limit" (current value: "12288", data source: default value)
                         Size (in bytes) of "phase 1" fragment sent for all large messages (must be >= 0 and <= eager_limit)
                MCA btl: parameter "btl_openib_eager_limit" (current value: "12288", data source: default value)
                         Maximum size (in bytes) of "short" messages (must be >= 1).
                MCA btl: parameter "btl_openib_max_send_size" (current value: "65536", data source: default value)
                         Maximum size (in bytes) of a single "phase 2" fragment of a long message when using the pipeline protocol (must be >=
                         1)
                MCA btl: parameter "btl_openib_rdma_pipeline_send_length" (current value: "1048576", data source: default value)
                         Length of the "phase 2" portion of a large message (in bytes) when using the pipeline protocol. This part of the
                         message will be split into fragments of size max_send_size and sent using send/receive semantics (must be >= 0; only
                         relevant when the PUT flag is set)
                MCA btl: parameter "btl_openib_rdma_pipeline_frag_size" (current value: "1048576", data source: default value)
                         Maximum size (in bytes) of a single "phase 3" fragment from a long message when using the pipeline protocol. These
                         fragments will be sent using RDMA semantics (must be >= 1; only relevant when the PUT flag is set)
                MCA btl: parameter "btl_openib_min_rdma_pipeline_size" (current value: "262144", data source: default value)
                         Messages smaller than this size (in bytes) will not use the RDMA pipeline protocol. Instead, they will be split into
                         fragments of max_send_size and sent using send/receive semantics (must be >=0, and is automatically adjusted up to at
                         least (eager_limit+btl_rdma_pipeline_send_length); only relevant when the PUT flag is set)
                MCA btl: parameter "btl_openib_bandwidth" (current value: "800", data source: default value)
                         Approximate maximum bandwidth of interconnect(must be >= 1)
                MCA btl: parameter "btl_openib_latency" (current value: "10", data source: default value)
                         Approximate latency of interconnect (must be >= 0)
                MCA btl: parameter "btl_openib_receive_queues" (current value:
                         "P,128,256,192,128:S,2048,256,128,32:S,12288,256,128,32:S,65536,256,128,32", data source: default value)
                         Colon-delimited, comma delimited list of receive queues: P,4096,8,6,4:P,32768,8,6,4
                MCA btl: parameter "btl_openib_if_include" (current value: <none>, data source: default value)
                         Comma-delimited list of devices/ports to be used (e.g. "mthca0,mthca1:2"; empty value means to use all ports found).
                         Mutually exclusive with btl_openib_if_exclude.
                MCA btl: parameter "btl_openib_if_exclude" (current value: <none>, data source: default value)
                         Comma-delimited list of device/ports to be excluded (empty value means to not exclude any ports). Mutually exclusive
                         with btl_openib_if_include.
                MCA btl: parameter "btl_openib_ipaddr_include" (current value: <none>, data source: default value)
                         Comma-delimited list of IP Addresses to be used (e.g. "192.168.1.0/24"). Mutually exclusive with
                         btl_openib_ipaddr_exclude.
                MCA btl: parameter "btl_openib_ipaddr_exclude" (current value: <none>, data source: default value)
                         Comma-delimited list of IP Addresses to be excluded (e.g. "192.168.1.0/24"). Mutually exclusive with
                         btl_openib_ipaddr_include.
                MCA btl: parameter "btl_openib_cpc_include" (current value: <none>, data source: default value)
                         Method used to select OpenFabrics connections (valid values: oob,xoob,rdmacm)
                MCA btl: parameter "btl_openib_cpc_exclude" (current value: <none>, data source: default value)
                        Method used to exclude OpenFabrics connections (valid values: oob,xoob,rdmacm)
                MCA btl: parameter "btl_openib_connect_oob_priority" (current value: "50", data source: default value)
                         The selection method priority for oob
                MCA btl: parameter "btl_openib_connect_xoob_priority" (current value: "60", data source: default value)
                         The selection method priority for xoob
                MCA btl: parameter "btl_openib_connect_rdmacm_priority" (current value: "30", data source: default value)
                         The selection method priority for rdma_cm
                MCA btl: parameter "btl_openib_connect_rdmacm_port" (current value: "0", data source: default value)
                         The selection method port for rdma_cm
                MCA btl: parameter "btl_openib_connect_rdmacm_resolve_timeout" (current value: "1000", data source: default value)
                         The timeout (in miliseconds) for address and route resolution
                MCA btl: parameter "btl_openib_connect_rdmacm_retry_count" (current value: "20", data source: default value)
                         Maximum number of times rdmacm will retry route resolution
                MCA btl: parameter "btl_openib_connect_rdmacm_reject_causes_connect_error" (current value: "0", data source: default value)
                         The drivers for some devices are buggy such that an RDMA REJECT action may result in a CONNECT_ERROR event instead of
                         a REJECTED event. Setting this MCA parameter to true tells Open MPI to treat CONNECT_ERROR events on connections
                         where a REJECT is expected as a REJECT (default: false)
                MCA btl: parameter "btl_openib_priority" (current value: "0", data source: default value)
                MCA btl: parameter "btl_base_warn_component_unused" (current value: "1", data source: default value)
                         This parameter is used to turn on warning messages when certain NICs are not used
_______________________________________________
users mailing list
users_at_[hidden]<mailto:users_at_[hidden]>
hxxp://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
users_at_[hidden]<mailto:users_at_[hidden]>
hxxp://www.open-mpi.org/mailman/listinfo.cgi/users