Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] openmpi over tcp
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2009-02-01 06:49:19


It looks like you compiled Open MPI against the QLogic PSM libraries
-- I see the PSM MTL plugin available. Here's some text from the OMPI
v1.3 README that clarifies the situation:

- There are two MPI network models available: "ob1" and "cm". "ob1"
   uses BTL ("Byte Transfer Layer") components for each supported
   network. "cm" uses MTL ("Matching Tranport Layer") components for
   each supported network.

   - "ob1" supports a variety of networks that can be used in
     combination with each other (per OS constraints; e.g., there are
     reports that the GM and OpenFabrics kernel drivers do not operate
     well together):
     - OpenFabrics: InfiniBand and iWARP
     - Loopback (send-to-self)
     - Myrinet: GM and MX
     - Portals
     - Quadrics Elan
     - Shared memory
     - TCP
     - SCTP
     - uDAPL

   - "cm" supports a smaller number of networks (and they cannot be
     used together), but may provide better better overall MPI
     performance:
     - Myrinet MX (not GM)
     - InfiniPath PSM
     - Portals

   Open MPI will, by default, choose to use "cm" when the InfiniPath
   PSM MTL can be used. Otherwise, OB1 will be used and the
   corresponding BTLs will be selected. Users can force the use of ob1
   or cm if desired by setting the "pml" MCA parameter at run-time:

     shell$ mpirun --mca pml ob1 ...
     or
     shell$ mpirun --mca pml cm ...

So to force TCP to be used, you need to force the use of the ob1 PML
and then force the use of the TCP BTL. Perhaps something like this:

     mpirun --mca pml ob1 --mca btl tcp,sm,self ...

On Jan 29, 2009, at 7:20 PM, Daniel De Marco wrote:

> Jeff,
>
> I put most of the info at:
> http://www.bartol.udel.edu/~ddm/ompi_debug.tgz
> The tar file contains the config.log, the ifconfig for the two nodes
> and
> the output of ompi_info --all.
>
> As I said I was running with:
> mpirun --mca btl tcp,self --prefix /share/apps/openmpi-1.3/gcc_ifort/
> --machinefile machines -np 2 ./osu_latency
> and I also tried adding -mca btl_tcp_if_include eth0 to the options.
>
> Thanks for your help.
> Please let me know if you need some other info.
> Daniel.
>
>
> * Jeff Squyres <jsquyres_at_[hidden]> [01/29/2009 16:30]:
>> Can you send the full output described here (including all network
>> setup
>> stuff):
>>
>> http://www.open-mpi.org/community/help/
>>
>>
>> On Jan 29, 2009, at 3:18 PM, Daniel De Marco wrote:
>>
>>> Hi Ralph,
>>>
>>> * Ralph Castain <rhc_at_[hidden]> [01/29/2009 14:27]:
>>>> It is quite likely that you have IPoIB on your system. In that
>>>> case, the
>>>> TCP BTL will pickup that interface and use it.
>>>>
>>>> If you have a specific interface you want to use, try -mca
>>>> btl_tcp_if_include eth0 (or whatever that interface is). This
>>>> tell the
>>>> TCP
>>>> BTL to only use the specified interface, so it will either fail
>>>> (if that
>>>> interface isn't available or doesn't exist) or use only that one.
>>>
>>> no, I don't have IPoIB configured. I tried anyway your suggestion
>>> and I
>>> got the same results as before. The weird thing is that even if use
>>> -mca btl_tcp_if_include eth2
>>> where eth2 doesn't exist I get the same results...
>>>
>>> Thanks, Daniel.
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>> --
>> Jeff Squyres
>> Cisco Systems
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

-- 
Jeff Squyres
Cisco Systems