Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI users] ofa-default-subnet-gid
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2008-02-21 18:57:10


On Feb 21, 2008, at 12:36 PM, George Bosilca wrote:

> Here are the MCA parameters that you can use:
>
> MCA btl: parameter "btl_openib_if_include" (current value:
> <none>)
> Comma-delimited list of HCAs/ports to be used (e.g.
> "mthca0,mthca1:2"; empty value means to use all
> ports found).
> Mutually exclusive with btl_openib_if_exclude.
> MCA btl: parameter "btl_openib_if_exclude" (current value:
> <none>)
> Comma-delimited list of HCAs/ports to be excluded
> (empty value
> means to not exclude any ports). Mutually
> exclusive with
> btl_openib_if_include.

These parameters are [upcoming] v1.3 only -- they do not exist in the
v1.2 series.

(more below)

> On Feb 21, 2008, at 2:45 PM, Bill Wichser wrote:

>> In trying to get openmpi up and running on a new cluster, I came
>> across
>> this error about having both of my IB switches set to the same
>> subnet-gid. Snooping around on my hosts which run the opensm
>> daemon, I
>> indeed found this to be the case in the /var/log/osm-ib[0-1].log
>> files,
>> giving up finding it with ibstat which showed these values to be
>> different, at least the second part of the GID.
>>
>> Before I try and pursue how to actually change this value for the
>> opensm
>> daemon, I do have a question.
>>
>> Since both of my hosts are connected to each switch, how am I to
>> instruct openmpi to use port0? I'm trying to use port0 as the MPI
>> network and port1 as the storage network. Is there something that I
>> need to add someplace forcing connections only to some default-
>> subnet-gid?

The v1.3 series will have the parameters that George mentioned above;
those give you fine-grained control about which HCAs and ports you are
using.

In the v1.2 series, you cannot explicitly control which hcas/ports you
are using. Instead, you can only limit the *number* of active ports
that Open MPI will use:

              MCA btl: parameter "btl_openib_max_btls" (current value:
"-1")
                       Maximum number of HCA ports to use (-1 = use all
                       available, otherwise must be >= 1)

Open MPI starts with the first port on the first interface and goes
upward until it finds max_btls of active ports. This is admittedly
imperfect, but it was only somewhat recently that someone asked for
explicit control over which hcas/ports to use. Sorry... :-\

-- 
Jeff Squyres
Cisco Systems