Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Question about '--mca btl tcp,self'
From: Jeff Squyres (jsquyres) (jsquyres_at_[hidden])
Date: 2014-03-17 13:09:49


On Mar 17, 2014, at 12:37 PM, Gus Correa <gus_at_[hidden]> wrote:

> In other words, does OOB pick the interface associated to the IP address
> that resolves the specific node name, or does OOB have its own will and
> picks whatever interface it wants?

I'll let Ralph contribute the detail here, but it's basically the latter: the OOB has its own will and picks whatever interface it wants.

But keep in mind that this is true for ALL OMPI communications (including MPI communications): the hostfile is unrelated to what interfaces are used.

Early MPI implementations back in the 90's overloaded the use of the hostfile with which network interfaces were used. Open MPI has never used that approach: we have always used the hostfile (and --host, etc.) as simply a mechanism to specify which servers/compute nodes/whatever on which to run. Selection of interfaces to use for control messages and MPI messages are determined separately.

> In a node outfitted with more than one Inifinband interface,
> can one choose which one OMPI is going to use (say, if one wants to
> reserve the other IB interface for IO)?

Yes. Each BTL typically has it's own MCA param for this kind of thing. You might want to troll through ompi_info output to see if there's anything of interest to you. For example:

  ompi_info --param btl openib --level 9

(the "--level 9" option is new somewhere during the 1.7.x series; it will cause a syntax error in the 1.6 series)

will show you all the MCA params for the openib BTL. The one you want for the openib BTL is:

    mpirun --mca btl_openib_if_include <interfaces>

With the usnic BTL, we allow you to specify interfaces via two different kinds of values:

    mpirun --mca btl_usnic_if_include <interfaces>

where interfaces can be:

    usnic_X (e.g., usnic_0)
    CIDR network address (e.g., 192.168.0.0/16)
    
>> Also, note that you seem to have missed a BTL: sm (shared memory).
> sm is the preferred BTL to use for same-server communication.
>
> This may be because several FAQs skip the sm BTL, even when it would
> be an appropriate/recommended choice to include in the BTL list.
> For instance:
>
> http://www.open-mpi.org/faq/?category=all#selecting-components

This one seems to be ok. I think the item you're referring to in that entry is an example of the ^ negation operator.

> http://www.open-mpi.org/faq/?category=all#tcp-selection

Fixed. Thanks!

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/