Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Can't use tcp instead of openib/infinipath
From: Bill Broadley (bill_at_[hidden])
Date: 2008-07-23 17:35:34

Jeff Squyres wrote:
> Sorry for the delay in replying.
> What exactly is the relay program timing? Can you run a standard
> benchmark like NetPIPE, perchance? (

It gives very similar numbers to osu_latency. Turns out the mca btl seems to
be completely ignored, I.e.:
[bill_at_compute-0-0 relay]$ mpirun -np 2 -mca btl foo -machinefile m ./relay 1
compute-0-0.local compute-0-1.local
size= 1, 131072 hops, 2 nodes in 0.266 sec ( 2.027 us/hop) 1928 KB/sec

mpirun -np 2 -mca btl foo -machinefile m \
# OSU MPI Bandwidth Test v3.0
# Size Bandwidth (MB/s)
1 2.40

My understanding is that -mca btl foo should fail since there isn't a
transport layer called foo.

[bill_at_compute-0-0 relay]$ which mpirun

ldd ./relay => /lib64/ (0x00002aaaaacc7000) => /usr/mpi/gcc/openmpi-1.2.6/lib64/ (0x00002aaaaaf4a000) => /usr/mpi/gcc/openmpi-1.2.6/lib64/
(0x00002aaaab1d8000) => /usr/mpi/gcc/openmpi-1.2.6/lib64/
(0x00002aaaab433000) => /lib64/ (0x00002aaaab692000) => /lib64/ (0x00002aaaab896000) => /lib64/ (0x00002aaaabaaf000) => /lib64/ (0x00002aaaabcb2000) => /lib64/ (0x00002aaaabecc000)
        /lib64/ (0x00002aaaaaaab000)

So OFED-1.3.1 (or an openmpi build from source) ./ works with TCP,
but not infinipath (because of a missing psm library). All the "-mca btl"
functionality works as expected.

OFED-1.3.1 (or an openmpi build from source) when I add "--with-psm" works
with infinipath, but all -mca parameters are ignored. Is there a way to get
openmpi working with infinipath without the psm library? Or a suggestion on
how to get the -mca functionality working?