Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] RFC: Move of ompi_bitmap_t
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2009-02-03 18:31:16


Ah; I thought this data structure was used for the fortran MPI object
handles as well, but looking at the code now, I see that
opal_pointer_array's are used for that. The only Fortran place it is
used is for attributes.

I see ompi_bitmap_t used for attributes, BTL reachability
computations, and the crcp. None of these strike me as performance
sensitive at all. I see it used in the tuned collectives but I agree
that one additional compare probably doesn't matter (sure, if we can
remove it, that's great -- but in the larger scheme of things, we're
going to pay much more in latency for collectives than one comparison
will cost).

I see Ralph's argument about the max Fortran value being defined in
opal_config.h, but remember that that's only a side-effect of how
Autoconf works. If Autoconf had allowed us, we would have had 3 truly
different files seeded with different #defines from configure:
opal_config.h, orte_config.h, and ompi_config.h (right now we dump
everything in opal_config.h the others essentially wholly include
opal_config.h and add a few more values/defines -- we don't have a
clear separation of each layer's results from configure).

I think setting some reasonable max size for ompi_bitmap_t is fine
with a new API call to re-define it if desired would be fine (e.g.,
the MPI layer can call it with the max fortran value to ensure that it
has the size that it needs for attributes). But if someone wants to
re-code the whole thing to have a definite max size (i.e., not re-size
if you set a bit that doesn't yet exist), go to it. I don't really
care. It strikes me that there's more important stuff to do in our
code base than to optimize our glue bitmap code, though. :-)

Ok, I'm now done talking about this. :-)

On Feb 3, 2009, at 4:45 PM, George Bosilca wrote:

> These places are easy to find and track. I did it on the ORTE layer,
> and in this context the bitmap is _NOT_ required to grow as all
> bitmaps are initialize with the number of processes in the jobid. In
> the OMPI layer there are few places using the bitmap:
>
> - the hierarch collective. There the bitmap is initialized with the
> size of the communicator, so it will _NEVER_ get expanded.
>
> - in the PML (DR and OB1). Again the bitmap is initialized using the
> number of processes, so it will _NEVER_ get expanded.
>
> - in the attributes. This is the only place where the bitmap might
> expand. However, as the current implementation is not thread safe
> and as this call is outside the critical path, we can modify it in
> order to expand the bitmap manually.
>
> So, it appears that we don't really take advantage of the original
> design for the bitmap. It might be time to revise it ...
>
> george.
>
> On Feb 3, 2009, at 15:30 , Jeff Squyres wrote:
>
>> On Feb 3, 2009, at 3:24 PM, George Bosilca wrote:
>>
>>> In the current bitmap implementation every time we set or check a
>>> bit we have to compute the index of the char where this bit is set
>>> and the relative position from the beginning of char. This
>>> requires two _VERY_ expensive operations: a division and a modulo.
>>> Compared with the cost of these two operation a quick test for a
>>> max bit is irrelevant.
>>>
>>> In fact I think the safety limit if good for most cases. How about
>>> having the max bit to the limit used to initialize the bitmap? We
>>> can add a call to extend the bitmap in case some layer really need
>>> to extend it, but restrict all others layers to the number of bits
>>> requested when the bitmap is initialized.
>>
>> The problem with this is that the original design expands the
>> bitmap whenever you try to set a bit that doesn't yet exist. So
>> you'd need to track down every place in the code that exercises
>> this assumption.
>>
>> You could set a max size if you want to (e.g., assuming you'll
>> never have more than some_large_value of fortran handles [probably
>> considerably less than the number of Fortran integers available],
>> or somesuch).
>>
>> --
>> Jeff Squyres
>> Cisco Systems
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Jeff Squyres
Cisco Systems