Tim Mattox wrote:
> For your runs with Open MPI over InfiniBand, try using openib,sm,self
> for the BTL setting, so that shared memory communications are used
> within a node. It would give us another datapoint to help diagnose
> the problem. As for other things we would need to help diagnose the
> problem, please follow the advice on this FAQ entry, and the help page:
> http://www.open-mpi.org/faq/?category=openfabrics#ofa-troubleshoot
> http://www.open-mpi.org/community/help/
>
Dear Tim,
thank you for this pointer.
1) Ofed: It's 1.2.5, from the OpenFabrics website
2) Linux version: scientific linux (RH enterprise remaster) v. 4.2,
kernel 2.6.9-55.0.12.ELsmp
3) Subnet manager: OpenSM
4)ibv_devinfo
hca_id: mthca0
fw_ver: 1.0.800
node_guid: 0002:c902:0022:b398
sys_image_guid: 0002:c902:0022:b39b
vendor_id: 0x02c9
vendor_part_id: 25204
hw_ver: 0xA0
board_id: MT_03B0120002
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 2048 (4)
active_mtu: 2048 (4)
sm_lid: 9
port_lid: 97
port_lmc: 0x00
(no node is different from the others, as far as the problem is concerned)
5) ifconfig:
eth0 Link encap:Ethernet HWaddr 00:17:31:E3:89:4A
inet addr:10.0.0.12 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::217:31ff:fee3:894a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:23348585 errors:0 dropped:0 overruns:0 frame:0
TX packets:17247486 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:19410724189 (18.0 GiB) TX bytes:14981325997 (13.9 GiB)
Interrupt:209
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:5088 errors:0 dropped:0 overruns:0 frame:0
TX packets:5088 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2468843 (2.3 MiB) TX bytes:2468843 (2.3 MiB)
6) ulimit -l
8388608
(this is more than the physical memory on the node)
7) output of ompi_info attached (I have tried also earlier releases)
8) description of the problem: a program seems to communicate correctly
over the TCP network, but not over the ifiniband network. The program is
structured in such a way that if the communication does not happen, a
loop become infinite. So there is no error message, just a program
entering an infinite loop.
The command line used are:
The command line I use is
mpirun -mca btl openib,sm,self <executable>
(with openib replaced by tcp in the case of communication over ethernet).
I could include the path and the value of the variable LD_LIBRARY_PATH,
but it won't tell too much, since the installation directory is
non-standard (/opt/ompi128-intel/bin for the path and
/opt/ompi128-intel/lib for the libs).
I hope to have provided all the required info, if you need more or some
of them in more detail, please let me know.
Many thanks,
Biagio Lucini
Open MPI: 1.2.8
Open MPI SVN revision: r19718
Open RTE: 1.2.8
Open RTE SVN revision: r19718
OPAL: 1.2.8
OPAL SVN revision: r19718
Prefix: /opt/ompi128-intel
Configured architecture: x86_64-unknown-linux-gnu
Configured by: root
Configured on: Tue Dec 23 12:33:51 GMT 2008
Configure host: master.cluster
Built by: root
Built on: Tue Dec 23 12:38:34 GMT 2008
Built host: master.cluster
C bindings: yes
C++ bindings: yes
Fortran77 bindings: yes (all)
Fortran90 bindings: yes
Fortran90 bindings size: small
C compiler: icc
C compiler absolute: /opt/intel/cce/9.1.045/bin/icc
C++ compiler: icpc
C++ compiler absolute: /opt/intel/cce/9.1.045/bin/icpc
Fortran77 compiler: ifort
Fortran77 compiler abs: /opt/intel/fce/9.1.040/bin/ifort
Fortran90 compiler: ifort
Fortran90 compiler abs: /opt/intel/fce/9.1.040/bin/ifort
C profiling: yes
C++ profiling: yes
Fortran77 profiling: yes
Fortran90 profiling: yes
C++ exceptions: no
Thread support: posix (mpi: no, progress: no)
Internal debug support: no
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
libltdl support: yes
Heterogeneous support: yes
mpirun default --prefix: no
MCA backtrace: execinfo (MCA v1.0, API v1.0, Component v1.2.8)
MCA memory: ptmalloc2 (MCA v1.0, API v1.0, Component v1.2.8)
MCA paffinity: linux (MCA v1.0, API v1.0, Component v1.2.8)
MCA maffinity: first_use (MCA v1.0, API v1.0, Component v1.2.8)
MCA maffinity: libnuma (MCA v1.0, API v1.0, Component v1.2.8)
MCA timer: linux (MCA v1.0, API v1.0, Component v1.2.8)
MCA installdirs: env (MCA v1.0, API v1.0, Component v1.2.8)
MCA installdirs: config (MCA v1.0, API v1.0, Component v1.2.8)
MCA allocator: basic (MCA v1.0, API v1.0, Component v1.0)
MCA allocator: bucket (MCA v1.0, API v1.0, Component v1.0)
MCA coll: basic (MCA v1.0, API v1.0, Component v1.2.8)
MCA coll: self (MCA v1.0, API v1.0, Component v1.2.8)
MCA coll: sm (MCA v1.0, API v1.0, Component v1.2.8)
MCA coll: tuned (MCA v1.0, API v1.0, Component v1.2.8)
MCA io: romio (MCA v1.0, API v1.0, Component v1.2.8)
MCA mpool: rdma (MCA v1.0, API v1.0, Component v1.2.8)
MCA mpool: sm (MCA v1.0, API v1.0, Component v1.2.8)
MCA pml: cm (MCA v1.0, API v1.0, Component v1.2.8)
MCA pml: ob1 (MCA v1.0, API v1.0, Component v1.2.8)
MCA bml: r2 (MCA v1.0, API v1.0, Component v1.2.8)
MCA rcache: vma (MCA v1.0, API v1.0, Component v1.2.8)
MCA btl: openib (MCA v1.0, API v1.0.1, Component v1.2.8)
MCA btl: self (MCA v1.0, API v1.0.1, Component v1.2.8)
MCA btl: sm (MCA v1.0, API v1.0.1, Component v1.2.8)
MCA btl: tcp (MCA v1.0, API v1.0.1, Component v1.0)
MCA topo: unity (MCA v1.0, API v1.0, Component v1.2.8)
MCA osc: pt2pt (MCA v1.0, API v1.0, Component v1.2.8)
MCA errmgr: hnp (MCA v1.0, API v1.3, Component v1.2.8)
MCA errmgr: orted (MCA v1.0, API v1.3, Component v1.2.8)
MCA errmgr: proxy (MCA v1.0, API v1.3, Component v1.2.8)
MCA gpr: null (MCA v1.0, API v1.0, Component v1.2.8)
MCA gpr: proxy (MCA v1.0, API v1.0, Component v1.2.8)
MCA gpr: replica (MCA v1.0, API v1.0, Component v1.2.8)
MCA iof: proxy (MCA v1.0, API v1.0, Component v1.2.8)
MCA iof: svc (MCA v1.0, API v1.0, Component v1.2.8)
MCA ns: proxy (MCA v1.0, API v2.0, Component v1.2.8)
MCA ns: replica (MCA v1.0, API v2.0, Component v1.2.8)
MCA oob: tcp (MCA v1.0, API v1.0, Component v1.0)
MCA ras: dash_host (MCA v1.0, API v1.3, Component v1.2.8)
MCA ras: gridengine (MCA v1.0, API v1.3, Component v1.2.8)
MCA ras: localhost (MCA v1.0, API v1.3, Component v1.2.8)
MCA ras: slurm (MCA v1.0, API v1.3, Component v1.2.8)
MCA rds: hostfile (MCA v1.0, API v1.3, Component v1.2.8)
MCA rds: proxy (MCA v1.0, API v1.3, Component v1.2.8)
MCA rds: resfile (MCA v1.0, API v1.3, Component v1.2.8)
MCA rmaps: round_robin (MCA v1.0, API v1.3, Component v1.2.8)
MCA rmgr: proxy (MCA v1.0, API v2.0, Component v1.2.8)
MCA rmgr: urm (MCA v1.0, API v2.0, Component v1.2.8)
MCA rml: oob (MCA v1.0, API v1.0, Component v1.2.8)
MCA pls: gridengine (MCA v1.0, API v1.3, Component v1.2.8)
MCA pls: proxy (MCA v1.0, API v1.3, Component v1.2.8)
MCA pls: rsh (MCA v1.0, API v1.3, Component v1.2.8)
MCA pls: slurm (MCA v1.0, API v1.3, Component v1.2.8)
MCA sds: env (MCA v1.0, API v1.0, Component v1.2.8)
MCA sds: pipe (MCA v1.0, API v1.0, Component v1.2.8)
MCA sds: seed (MCA v1.0, API v1.0, Component v1.2.8)
MCA sds: singleton (MCA v1.0, API v1.0, Component v1.2.8)
MCA sds: slurm (MCA v1.0, API v1.0, Component v1.2.8)
|
![[Flag]](../../../../../community/mirrors/flag-us.gif){kind=small}