Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Inherent limit on #communicators?
From: Edgar Gabriel (gabriel_at_[hidden])
Date: 2009-04-30 14:44:59


Brian W. Barrett wrote:
> When we added the CM PML, we added a pml_max_contextid field to the PML
> structure, which is the max size cid the PML can handle (because the
> matching interfaces don't allow 32 bits to be used for the cid. At the
> same time, the max cid for OB1 was shrunk significantly, so that the
> header on a short message would be packed tightly with no alignment
> padding.
>
> At the time, we believed 32k simultaneous communicators was plenty, and
> that CIDs were reused (we checked, I'm pretty sure). It sounds like
> someone removed the CID reuse code, which seems rather bad to me.

yes, we added the block algorithm. Not reusing a CID actually doesn't
bite me as that dramatic, and I am still not sure and convinced about
that:-) We do not have an empty array or something like that, its just a
number.

The reason for the block algorithm was that the performance of our
communicator creation code sucked, and the cid allocation was one
portion of that. We used to require *at least* 4 collective operations
per communicator creation at that time. We are now potentially down to
0, among others thanks to the block algorithm.

However, let me think about reusing entire blocks, its probably doable
just requires a little more bookkeeping...

> There
> have to be unused CIDs in Ralph's example - is there a way to fallback
> out of the block algorithm when it can't find a new CID and find one it
> can reuse? Other than setting the multi-threaded case back on, that is?

remember that its not the communicator id allocation that is failing at
this point, so the question is do we have to 'validate' a cid with the
pml before we declare it to be ok?

Thanks
Edgar

>
> Brian
>
> On Thu, 30 Apr 2009, Edgar Gabriel wrote:
>
>> cid's are in fact not recycled in the block algorithm. The problem is
>> that comm_free is not collective, so you can not make any assumptions
>> whether other procs have also released that communicator.
>>
>>
>> But nevertheless, a cid in the communicator structure is a uint32_t,
>> so it should not hit the 16k limit there yet. this is not new, so if
>> there is a discrepancy between what the comm structure assumes that a
>> cid is and what the pml assumes, than this was in the code since the
>> very first days of Open MPI...
>>
>> Thanks
>> Edgar
>>
>> Brian W. Barrett wrote:
>>> On Thu, 30 Apr 2009, Ralph Castain wrote:
>>>
>>>> We seem to have hit a problem here - it looks like we are seeing a
>>>> built-in limit on the number of communicators one can create in a
>>>> program. The program basically does a loop, calling MPI_Comm_split each
>>>> time through the loop to create a sub-communicator, does a reduce
>>>> operation on the members of the sub-communicator, and then calls
>>>> MPI_Comm_free to release it (this is a minimized reproducer for the
>>>> real
>>>> code). After 64k times through the loop, the program fails.
>>>>
>>>> This looks remarkably like a 16-bit index that hits a max value and
>>>> then
>>>> blocks.
>>>>
>>>> I have looked at the communicator code, but I don't immediately see
>>>> such
>>>> a field. Is anyone aware of some other place where we would have a
>>>> limit
>>>> that would cause this problem?
>>>
>>> There's a maximum of 32768 communicator ids when using OB1 (each PML
>>> can set the max contextid, although the communicator code is the part
>>> that actually assigns a cid). Assuming that comm_free is actually
>>> properly called, there should be plenty of cids available for that
>>> pattern. However, I'm not sure I understand the block algorithm
>>> someone added to cid allocation - I'd have to guess that there's
>>> something funny with that routine and cids aren't being recycled
>>> properly.
>>>
>>> Brian
>>> _______________________________________________
>>> devel mailing list
>>> devel_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab      http://pstl.cs.uh.edu
Department of Computer Science          University of Houston
Philip G. Hoffman Hall, Room 524        Houston, TX-77204, USA
Tel: +1 (713) 743-3857                  Fax: +1 (713) 743-3335