Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] RFC: changes to modex
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2008-04-02 20:41:14

On Apr 2, 2008, at 4:12 PM, Gleb Natapov wrote:
>>> I can specify
>>> different openib_if_include values for different procs on the same
>>> host.
>> I know you *can*, but it is certainly uncommon. The common case is
> Uncommon - yes, but do you what to make it unsupported?

No, there's no need for that.

>> that it's the same for all procs on all hosts. I guess there's a few
>> cases:
>> 1. homogeneous include/exclude, no carto: send all in node info; no
>> proc info
>> 2. homogeneous include/exclude, carto is used: send all ports in node
>> info; send index in proc info for which node info port index it
>> will use
> This may actually increase modex size. Think about two procs using two
> different hcas. We'll send all the data we send today + indexes.

It'll increase it compared to the optimization that we're about to
make. But it will certainly be a large decrease compared to what
we're doing today (see the spreadsheet that I sent last week).

Indeed, we can even put in the optimization that if there's only one
process on a host, it can only publish the ports that it will use (and
therefore there's no need for the proc data).

>> 3. heterogeneous include/exclude, no cart: need user to tell us that
>> this situation exists (e.g., use another MCA param), but then is same
>> as #2
>> 4. heterogeneous include/exclude, cart is used, same as #3
>> Right?
> Looks like it. FWIW I don't like the idea to code all those special
> cases. The way it works now I can be pretty sure that any crazy setup
> I'll come up with will work.

And so it will with the new scheme. The only place it won't work is
if the user specifies a heterogeneous include/exclude (i.e., we'll
require that the user tells us when they do that), which nobody does.

I guess I don't see the problem...?

> By the way how much data are moved during modex stage? What if modex
> will use compression?

The spreadsheet I listed was just the openib part of the modex, and it
was fairly hefty. I have no idea how well (or not) it would compress.

Jeff Squyres
Cisco Systems