I'm afraid I'm confused by your question, so please clarify: the hostnames that have ethernet interfaces do NOT have infiniband interfaces on them? If so, why would you expect OMPI to use infiniband when you execute on those hosts? Or do they contain an infiniband interface in addition to the ethernet interface?


On Mar 14, 2010, at 4:47 PM, nad labassi wrote:

Hi,

My problem is: I installed openmpi 1.2.9, with mvapi support, but the
execution is done on ethernet, unless I use ipoib.

I built openmpi with the command:

./configure CC=/home/pgi/linux86-64/6.2/bin/pgcc
CXX=/home/pgi/linux86-64/6.2/bin/pgCC 
FC=/home/pgi/linux86-64/6.2/bin/pgf90 
--without-tm --without-gm --without-mx --with-sge --without-openib 
--with-mvapi=/usr/mellanox --with-mvapi-libdir=/usr/lib64 
--with-threads=posix
--prefix=/home/openmpi-1.2.9_mvapi

I run the program with the commande:

/home/openmpi-1.2.9_mvapi/bin/mpirun --mca btl mvapi,self -n 8
-machinefile /home/myaccount/machines
/home/myaccount/mpihello_openmpi-1.2.9.exe

When the machines file contains hostnames for ethernet interfaces,
the execution is never done on infiniband

When the machines file contains hostnames for ib interfaces (ipoib),
then the execution is done on infiniband.

I want the execution to be done on infiniband, while the machines
file contains hostnames for ethernet interfaces.


 ompi_info --param btl mvapi outputs are:

                 MCA btl: parameter "btl_base_debug" (current value:
"0")
                          If btl_base_debug is 1 standard debug is
output, if > 1 verbose debug is output
                 MCA btl: parameter "btl" (current value: <none>)
                          Default selection set of components for the
btl framework (<none> means "use all
 components that can be found")
                 MCA btl: parameter "btl_base_verbose" (current
value: "0")
                          Verbosity level for the btl framework (0 =
no verbosity)
                 MCA btl: parameter "btl_mvapi_max_btls" (current
value: "4")
                          maximum number of HCAs/ports to use
                 MCA btl: parameter "btl_mvapi_free_list_num"
(current value: "8")
                          initial size of free lists
                 MCA btl: parameter "btl_mvapi_free_list_max"
(current value: "-1")
                          maximum size of free lists
                 MCA btl: parameter "btl_mvapi_free_list_inc"
(current value: "32")
                          increment size of free lists
                 MCA btl: parameter "btl_mvapi_mpool" (current value:
"rdma")
                          name of the memory pool to be used
                 MCA btl: parameter "btl_mvapi_reg_mru_len" (current
value: "16")
                          length of the registration cache most
recently used list
                 MCA btl: parameter "btl_mvapi_use_srq" (current
value: "0")
                          if 1 use the IB shared receive queue to
post receive descriptors
                 MCA btl: parameter "btl_mvapi_ib_cq_size" (current
value: "10000")
                          size of the IB completion queue
                 MCA btl: parameter "btl_mvapi_ib_sg_list_size"
(current value: "1")
                          size of IB segment list
                 MCA btl: parameter "btl_mvapi_ib_pkey_ix" (current
value: "0")
                          IB pkey index
                 MCA btl: parameter "btl_mvapi_ib_psn" (current
value: "0")
                          IB Packet sequence starting number
                 MCA btl: parameter "btl_mvapi_ib_qp_ous_rd_atom"
(current value: "4")
                          IB outstanding atomic reads

                 MCA btl: parameter "btl_mvapi_ib_mtu" (current
value: "3")
                          IB MTU, in bytes.  Valid values are: 1=256
bytes, 2=512 bytes, 3=1024 bytes, 4=2
048 bytes, 5=4096 bytes.
                 MCA btl: parameter "btl_mvapi_ib_min_rnr_timer"
(current value: "5")
                          IB min rnr timer
                 MCA btl: parameter "btl_mvapi_ib_timeout" (current
value: "10")
                          IB transmit timeout
                 MCA btl: parameter "btl_mvapi_ib_retry_count"
(current value: "7")
                          IB transmit retry count
                 MCA btl: parameter "btl_mvapi_ib_rnr_retry" (current
value: "7")
                          IB rnr retry
                 MCA btl: parameter "btl_mvapi_ib_max_rdma_dst_ops"
(current value: "4")
                          IB max rdma destination operations
                 MCA btl: parameter "btl_mvapi_ib_service_level"
(current value: "0")
                          IB service level
                 MCA btl: parameter "btl_mvapi_ib_static_rate"
(current value: "0")
(current value: "0")
                          IB static rate
                 MCA btl: parameter "btl_mvapi_ib_src_path_bits"
(current value: "0")
                          IB source path bits
                 MCA btl: parameter "btl_mvapi_rd_num" (current
value: "8")
                          number of receive descriptors to post to a
QP
                 MCA btl: parameter "btl_mvapi_rd_low" (current
value: "6")
                          low water mark before reposting occurs
                 MCA btl: parameter "btl_mvapi_rd_win" (current
value: "4")
                          window size at which generate explicity
credit message
                 MCA btl: parameter "btl_mvapi_srq_rd_max" (current
value: "1000")
                          Maximum number of receive descriptors
posted per SRQ.

                 MCA btl: parameter "btl_mvapi_srq_rd_per_peer"
(current value: "16")
                          receive descriptors posted per peer, SRQ
mode only
                 MCA btl: parameter "btl_mvapi_srq_sd_max" (current
value: "8")
                          Maximum number of send descriptors posted
per process
                 MCA btl: parameter "btl_mvapi_use_eager_rdma"
(current value: "1")
                          user RDMA for eager messages
                 MCA btl: parameter "btl_mvapi_eager_rdma_threshold"
(current value: "16")
                          Open rdma channel for eager messages after
this number of messages received from
 peer (zero to disable)
                 MCA btl: parameter "btl_mvapi_max_eager_rdma"

(current value: "16")
                          Maximum number of eager RDMA connections
                 MCA btl: parameter "btl_mvapi_eager_rdma_num"
(current value: "16")
                          Number of RDMA buffers for eager messages
                 MCA btl: parameter "btl_mvapi_exclusivity" (current
value: "1024")
                          BTL exclusivity
                 MCA btl: parameter "btl_mvapi_eager_limit" (current
value: "12288")
                          eager send limit
                 MCA btl: parameter "btl_mvapi_min_send_size"
(current value: "32768")
                          minimum send size
                 MCA btl: parameter "btl_mvapi_max_send_size"
(current value: "65536")
                          maximum send size
                 MCA btl: parameter "btl_mvapi_min_rdma_size"
(current value: "1048576")
                          minimum rdma size
                 MCA btl: parameter "btl_mvapi_max_rdma_size"
(current value: "1048576")
                          maximium rdma size
                 MCA btl: parameter "btl_mvapi_flags" (current value:
"6")
                          BTL flags, SEND=1, PUT=2, GET=4
                 MCA btl: parameter "btl_mvapi_bandwidth" (current
value: "800")
                          Approximate maximum bandwidth of
interconnect
                 MCA btl: parameter "btl_mvapi_priority" (current
value: "0")
                 MCA btl: parameter "btl_base_warn_component_unused"
(current value: "1")
                          This parameter is used to turn on warning
messages when certain NICs are not use


Thanks a lot,

Nadjet.





Hotmail : une messagerie fiable avec une protection anti-spam performante Inscrivez-vous_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users