On Apr 2, 2008, at 4:12 PM, Gleb Natapov wrote:
>>> I can specify
>>> different openib_if_include values for different procs on the same
>> I know you *can*, but it is certainly uncommon. The common case is
> Uncommon - yes, but do you what to make it unsupported?
No, there's no need for that.
>> that it's the same for all procs on all hosts. I guess there's a few
>> 1. homogeneous include/exclude, no carto: send all in node info; no
>> proc info
>> 2. homogeneous include/exclude, carto is used: send all ports in node
>> info; send index in proc info for which node info port index it
>> will use
> This may actually increase modex size. Think about two procs using two
> different hcas. We'll send all the data we send today + indexes.
It'll increase it compared to the optimization that we're about to
make. But it will certainly be a large decrease compared to what
we're doing today (see the spreadsheet that I sent last week).
Indeed, we can even put in the optimization that if there's only one
process on a host, it can only publish the ports that it will use (and
therefore there's no need for the proc data).
>> 3. heterogeneous include/exclude, no cart: need user to tell us that
>> this situation exists (e.g., use another MCA param), but then is same
>> as #2
>> 4. heterogeneous include/exclude, cart is used, same as #3
> Looks like it. FWIW I don't like the idea to code all those special
> cases. The way it works now I can be pretty sure that any crazy setup
> I'll come up with will work.
And so it will with the new scheme. The only place it won't work is
if the user specifies a heterogeneous include/exclude (i.e., we'll
require that the user tells us when they do that), which nobody does.
I guess I don't see the problem...?
> By the way how much data are moved during modex stage? What if modex
> will use compression?
The spreadsheet I listed was just the openib part of the modex, and it
was fairly hefty. I have no idea how well (or not) it would compress.