Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] ofa-default-subnet-gid
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2008-02-21 18:57:10

On Feb 21, 2008, at 12:36 PM, George Bosilca wrote:

> Here are the MCA parameters that you can use:
> MCA btl: parameter "btl_openib_if_include" (current value:
> <none>)
> Comma-delimited list of HCAs/ports to be used (e.g.
> "mthca0,mthca1:2"; empty value means to use all
> ports found).
> Mutually exclusive with btl_openib_if_exclude.
> MCA btl: parameter "btl_openib_if_exclude" (current value:
> <none>)
> Comma-delimited list of HCAs/ports to be excluded
> (empty value
> means to not exclude any ports). Mutually
> exclusive with
> btl_openib_if_include.

These parameters are [upcoming] v1.3 only -- they do not exist in the
v1.2 series.

(more below)

> On Feb 21, 2008, at 2:45 PM, Bill Wichser wrote:

>> In trying to get openmpi up and running on a new cluster, I came
>> across
>> this error about having both of my IB switches set to the same
>> subnet-gid. Snooping around on my hosts which run the opensm
>> daemon, I
>> indeed found this to be the case in the /var/log/osm-ib[0-1].log
>> files,
>> giving up finding it with ibstat which showed these values to be
>> different, at least the second part of the GID.
>> Before I try and pursue how to actually change this value for the
>> opensm
>> daemon, I do have a question.
>> Since both of my hosts are connected to each switch, how am I to
>> instruct openmpi to use port0? I'm trying to use port0 as the MPI
>> network and port1 as the storage network. Is there something that I
>> need to add someplace forcing connections only to some default-
>> subnet-gid?

The v1.3 series will have the parameters that George mentioned above;
those give you fine-grained control about which HCAs and ports you are

In the v1.2 series, you cannot explicitly control which hcas/ports you
are using. Instead, you can only limit the *number* of active ports
that Open MPI will use:

              MCA btl: parameter "btl_openib_max_btls" (current value:
                       Maximum number of HCA ports to use (-1 = use all
                       available, otherwise must be >= 1)

Open MPI starts with the first port on the first interface and goes
upward until it finds max_btls of active ports. This is admittedly
imperfect, but it was only somewhat recently that someone asked for
explicit control over which hcas/ports to use. Sorry... :-\

Jeff Squyres
Cisco Systems