Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Can't use tcp instead of openib/infinipath
From: Bill Broadley (bill_at_[hidden])
Date: 2008-07-23 17:35:34


Jeff Squyres wrote:
> Sorry for the delay in replying.
>
> What exactly is the relay program timing? Can you run a standard
> benchmark like NetPIPE, perchance? (http://www.scl.ameslab.gov/netpipe/)
>

It gives very similar numbers to osu_latency. Turns out the mca btl seems to
be completely ignored, I.e.:
[bill_at_compute-0-0 relay]$ mpirun -np 2 -mca btl foo -machinefile m ./relay 1
compute-0-0.local compute-0-1.local
size= 1, 131072 hops, 2 nodes in 0.266 sec ( 2.027 us/hop) 1928 KB/sec

Or:
mpirun -np 2 -mca btl foo -machinefile m \
/usr/mpi/gcc/openmpi-1.2.6/tests/osu_benchmarks-3.0/osu_bw
# OSU MPI Bandwidth Test v3.0
# Size Bandwidth (MB/s)
1 2.40
...

My understanding is that -mca btl foo should fail since there isn't a
transport layer called foo.

[bill_at_compute-0-0 relay]$ which mpirun
/usr/mpi/gcc/openmpi-1.2.6/bin/mpirun

ldd ./relay
        libm.so.6 => /lib64/libm.so.6 (0x00002aaaaacc7000)
        libmpi.so.0 => /usr/mpi/gcc/openmpi-1.2.6/lib64/libmpi.so.0 (0x00002aaaaaf4a000)
        libopen-rte.so.0 => /usr/mpi/gcc/openmpi-1.2.6/lib64/libopen-rte.so.0
(0x00002aaaab1d8000)
        libopen-pal.so.0 => /usr/mpi/gcc/openmpi-1.2.6/lib64/libopen-pal.so.0
(0x00002aaaab433000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab692000)
        libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab896000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaabaaf000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaabcb2000)
        libc.so.6 => /lib64/libc.so.6 (0x00002aaaabecc000)
        /lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)

So OFED-1.3.1 (or an openmpi build from source) ./install.pl works with TCP,
but not infinipath (because of a missing psm library). All the "-mca btl"
functionality works as expected.

OFED-1.3.1 (or an openmpi build from source) when I add "--with-psm" works
with infinipath, but all -mca parameters are ignored. Is there a way to get
openmpi working with infinipath without the psm library? Or a suggestion on
how to get the -mca functionality working?