Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] BTL add procs errors
From: Eugene Loh (eugene.loh_at_[hidden])
Date: 2010-06-02 16:40:44

Jeff Squyres wrote:

>Yes, I think the mmap code in the sm btl actually has a sync point inside add_procs that when the root allocs and sets up the area, it'll locally broadcast a "yes, we're good -- mmap attach and let's continue" or "bad things happened; sm btl is broke" message.
Yes, that's great. But my point was that (it used to be that) after
that point, processes would start eating chunks out of that shared area
and for large proc counts the last allocations would fail. (The size of
the shared area was poorly chosen and happened to be insufficient.) So,
despite the sync point you describe, some procs would succeed at
mca_btl_sm_add_procs() while others would not. This particular case is
now, I believe, resolved. It just seemed at the time like a case where
the upper layers were making assumptions that were inconsistent with
what the sm BTL was providing.

>But I am not confident about the other BTLs.
>On Jun 2, 2010, at 12:51 PM, Eugene Loh wrote:
>>George Bosilca wrote:
>>>We did assume that at least the errors are symmetric, i.e. if A fails
>>>to connect to B then B will fail when trying to connect to A.
>>I've not been following this thread closely, but thought I'd add a comment.
>>It used to be that the sm BTL could fail asymmetrically. A shared
>>memory could be allocated and processes start to allocate resources
>>within shared memory. At some point, the shared area would be
>>exhausted. So, some processes were set up to communicate to others, but
>>the others would not be able to communicate back via the same BTL. I
>>think this led to much brokenness. (E.g., how would a process return a
>>sm fragment to a sender?)
>>At this point, my recollection of those issues is very fuzzy.
>>In any case, I think those issues went away with the shared-memory work
>>I did a while back. The size of the area is now computed to be large
>>enough that each process's initial allocation would succeed.