When we added the CM PML, we added a pml_max_contextid field to the PML
structure, which is the max size cid the PML can handle (because the
matching interfaces don't allow 32 bits to be used for the cid. At the
same time, the max cid for OB1 was shrunk significantly, so that the
header on a short message would be packed tightly with no alignment
At the time, we believed 32k simultaneous communicators was plenty, and
that CIDs were reused (we checked, I'm pretty sure). It sounds like
someone removed the CID reuse code, which seems rather bad to me. There
have to be unused CIDs in Ralph's example - is there a way to fallback out
of the block algorithm when it can't find a new CID and find one it can
reuse? Other than setting the multi-threaded case back on, that is?
On Thu, 30 Apr 2009, Edgar Gabriel wrote:
> cid's are in fact not recycled in the block algorithm. The problem is that
> comm_free is not collective, so you can not make any assumptions whether
> other procs have also released that communicator.
> But nevertheless, a cid in the communicator structure is a uint32_t, so it
> should not hit the 16k limit there yet. this is not new, so if there is a
> discrepancy between what the comm structure assumes that a cid is and what
> the pml assumes, than this was in the code since the very first days of Open
> Brian W. Barrett wrote:
>> On Thu, 30 Apr 2009, Ralph Castain wrote:
>>> We seem to have hit a problem here - it looks like we are seeing a
>>> built-in limit on the number of communicators one can create in a
>>> program. The program basically does a loop, calling MPI_Comm_split each
>>> time through the loop to create a sub-communicator, does a reduce
>>> operation on the members of the sub-communicator, and then calls
>>> MPI_Comm_free to release it (this is a minimized reproducer for the real
>>> code). After 64k times through the loop, the program fails.
>>> This looks remarkably like a 16-bit index that hits a max value and then
>>> I have looked at the communicator code, but I don't immediately see such
>>> a field. Is anyone aware of some other place where we would have a limit
>>> that would cause this problem?
>> There's a maximum of 32768 communicator ids when using OB1 (each PML can
>> set the max contextid, although the communicator code is the part that
>> actually assigns a cid). Assuming that comm_free is actually properly
>> called, there should be plenty of cids available for that pattern. However,
>> I'm not sure I understand the block algorithm someone added to cid
>> allocation - I'd have to guess that there's something funny with that
>> routine and cids aren't being recycled properly.
>> devel mailing list