Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] tcp of openmpi-1.7.3 under our environment is very slow
From: Jeff Squyres (jsquyres) (jsquyres_at_[hidden])
Date: 2013-12-18 08:43:44


Can you re-run these tests with processor binding enabled?

On Dec 16, 2013, at 6:36 PM, tmishima_at_[hidden] wrote:

>
>
> Hi,
>
> I usually use infiniband network, where openmpi-1.7.3 and 1.6.5 works fine.
>
> The other days, I had a chance to use tcp network(1GbE) and I noticed that
> my application with openmpi-1.7.3 was quite slower than openmpi-1.6.5.
> So, I did OSU MPI Bandwidth Test v3.1.1 as shown below, which shows
> bandwidth for smaller size(< 1024) is very slow compared with 1.6.5.
> In addition, the latency for larger size( >65536 ) seems to be strange.
>
> Does this depend on our local environment or some mca parameter would be
> necesarry? I'm afraid that something is wrong with tcp of openmpi-1.7.3.
>
> openmpi-1.7.3:
>
> [mishima_at_node07 OMB-3.1.1]$ mpirun -np 2 -host node07,node08 -mca tbl
> ^openib osu_bw
> # OSU MPI Bandwidth Test v3.1.1
> # Size Bandwidth (MB/s)
> 1 0.00
> 2 0.01
> 4 0.01
> 8 0.03
> 16 0.05
> 32 0.10
> 64 0.32
> 128 0.37
> 256 0.87
> 512 5.97
> 1024 20.00
> 2048 182.87
> 4096 202.53
> 8192 215.14
> 16384 225.16
> 32768 228.58
> 65536 115.23
> 131072 198.24
> 262144 193.38
> 524288 233.03
> 1048576 227.31
> 2097152 233.07
> 4194304 233.25
>
> [mishima_at_node07 OMB-3.1.1]$ mpirun -np 2 -host node07,node08 -mca btl
> ^openib osu_latency
> # OSU MPI Latency Test v3.1.1
> # Size Latency (us)
> 0 19.23
> 1 19.57
> 2 19.52
> 4 19.88
> 8 20.44
> 16 20.38
> 32 20.78
> 64 21.14
> 128 21.75
> 256 23.20
> 512 26.12
> 1024 31.54
> 2048 41.72
> 4096 64.55
> 8192 107.52
> 16384 179.23
> 32768 251.58
> 65536 20689.68
> 131072 21179.79
> 262144 20168.56
> 524288 22984.83
> 1048576 25994.54
> 2097152 30929.55
> 4194304 38028.48
>
> openmpi-1.6.5:
>
> [mishima_at_node07 OMB-3.1.1]$ mpirun -np 2 -host node07,node08 -mca tbl
> ^openib osu_bw
> # OSU MPI Bandwidth Test v3.1.1
> # Size Bandwidth (MB/s)
> 1 0.22
> 2 0.45
> 4 0.89
> 8 1.77
> 16 3.57
> 32 7.15
> 64 14.28
> 128 28.58
> 256 57.17
> 512 96.44
> 1024 152.38
> 2048 182.84
> 4096 203.17
> 8192 215.13
> 16384 225.05
> 32768 100.58
> 65536 225.24
> 131072 182.92
> 262144 192.82
> 524288 212.92
> 1048576 233.35
> 2097152 233.72
> 4194304 233.89
>
> [mishima_at_node07 OMB-3.1.1]$ mpirun -np 2 -host node07,node08 -mca btl
> ^openib osu_latency
> # OSU MPI Latency Test v3.1.1
> # Size Latency (us)
> 0 17.24
> 1 17.30
> 2 17.29
> 4 17.30
> 8 24.32
> 16 17.24
> 32 17.80
> 64 17.91
> 128 19.08
> 256 20.81
> 512 22.83
> 1024 27.82
> 2048 39.54
> 4096 52.66
> 8192 97.70
> 16384 143.23
> 32768 215.02
> 65536 481.08
> 131072 800.64
> 262144 1475.12
> 524288 2698.62
> 1048576 4992.31
> 2097152 9558.96
> 4194304 20801.50
>
> Regards,
> Tetsuya Mishima
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/