Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Inherent limit on #communicators?
From: Edgar Gabriel (gabriel_at_[hidden])
Date: 2009-04-30 15:04:23


so I agree that we need to fix that, and we'll get a fix for that as
soon as possible. It still strikes me as wrong however to we have
fundamentally different types on two layers for the same 'item'.

I still think that going back to the original algorithm would be bad -
especially for an application that creates such a large number of
communicators potentially executed on a large number ( 1000s) of
processors. I'll look into how to reuse an entire block of communicator
cid respectively how to take the max_contextid into account.

Edgar

Brian W. Barrett wrote:
> On Thu, 30 Apr 2009, Edgar Gabriel wrote:
>
>> Brian W. Barrett wrote:
>>> When we added the CM PML, we added a pml_max_contextid field to the
>>> PML structure, which is the max size cid the PML can handle (because
>>> the matching interfaces don't allow 32 bits to be used for the cid.
>>> At the same time, the max cid for OB1 was shrunk significantly, so
>>> that the header on a short message would be packed tightly with no
>>> alignment padding.
>>>
>>> At the time, we believed 32k simultaneous communicators was plenty,
>>> and that CIDs were reused (we checked, I'm pretty sure). It sounds
>>> like someone removed the CID reuse code, which seems rather bad to me.
>>
>> yes, we added the block algorithm. Not reusing a CID actually doesn't
>> bite me as that dramatic, and I am still not sure and convinced about
>> that:-) We do not have an empty array or something like that, its just
>> a number.
>>
>> The reason for the block algorithm was that the performance of our
>> communicator creation code sucked, and the cid allocation was one
>> portion of that. We used to require *at least* 4 collective operations
>> per communicator creation at that time. We are now potentially down to
>> 0, among others thanks to the block algorithm.
>>
>> However, let me think about reusing entire blocks, its probably doable
>> just requires a little more bookkeeping...
>>
>>> There have to be unused CIDs in Ralph's example - is there a way to
>>> fallback out of the block algorithm when it can't find a new CID and
>>> find one it can reuse? Other than setting the multi-threaded case
>>> back on, that is?
>>
>> remember that its not the communicator id allocation that is failing
>> at this point, so the question is do we have to 'validate' a cid with
>> the pml before we declare it to be ok?
>
> well, that's only because the code's doing something it shouldn't. Have
> a look at comm_cid.c:185 - there's the check we added to the
> multi-threaded case (which was the only case when we added it). The cid
> generation should never generate a number larger than
> mca_pml.pml_max_contextid. I'm actually somewhat amazed this fails
> gracefully, as OB1 doesn't appear to check it got a valid cid in
> add_comm, which it should probably do.
>
> Looking at the differences between v1.2 and v1.3, the max_contextid code
> was already in v1.2 and OB1 was setting it to 32k. So the cid blocking
> code removed a rather critical feature and probably should be fixed or
> removed for v1.3. On Portals, I only get 8k cids, so not having reuse
> is a rather large problem.
>
> Brian
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab      http://pstl.cs.uh.edu
Department of Computer Science          University of Houston
Philip G. Hoffman Hall, Room 524        Houston, TX-77204, USA
Tel: +1 (713) 743-3857                  Fax: +1 (713) 743-3335