Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] can't disable infiniband communication
From: Sylvain Huet (sylvain.huet_at_[hidden])
Date: 2014-05-16 11:21:42


Dear all,

I am reinstalling a cluster where nodes are connected by
- 1 Gb ethernet interfaces
- 40 Gb infinband adapters
I installed OFED 3.12 on CentOS 6.5

I would like to be able to tell mpirun to use either the Gb ethernet
interfaces or the infinband adapters.

But, when I launch osu_bw with the following commands I always obtain
bandwidths of the infinibands links
(hostether is a file containing the IPs of the ethernet interfaces
IPs of ib adapters: 192.168.1.2,192.168.1.3
IPs of ethernet adapters: 192.168.70.12,192.168.70.13 )

mpirun --map-by node --mca btl ^openib --hostfile ./hostether -np 2
./osu_bw

mpirun --map-by node --mca btl_tcp_if_include eth0,eth1 --mca btl
tcp,self --hostfile ./hostether -np 2 ./osu_bw

mpirun --map-by node --mca btl_tcp_if_exclude ib0,ib1 --mca btl tcp,self
--hostfile ./hostether -np 2 ./osu_bw

mpirun --map-by node --mca btl_tcp_if_exclude
192.168.1.2,192.168.1.3,127.0.0.1/8,sppp --mca btl tcp,self --host
192.168.70.12,192.168.70.13 -np 2 ./osu_bw

All the previous commands lead to the following results with
either Open MPI 1.5.5, 1.7.5 or 1.8.1:

# OSU MPI Bandwidth Test (Version 2.3)
# Size Bandwidth (MB/s)
1 2.14
... ...
2097152 3015.64
4194304 3017.91

If I stop openibd I logically obtain these results:

# OSU MPI Bandwidth Test (Version 2.3)
# Size Bandwidth (MB/s)
1 0.19
... ...
2097152 117.63
4194304 117.66

Could someone give me some idea to investigate?

PS. I previouslly used this cluster with older versions
of CentOS and OFED and I was able to tell mpirun not to use
the infiniband adapters.

Best regards.