I also know little about that part of the code, but agree that does
seem weird. Seeing as we know how many local procs there are before we
get to this point, I would think we could be smart about our memory
pool size. You might not need to dive into the sm BTL to get the info
you need - if all you need is how many procs are local, that can be
obtained fairly easily.
Be happy to contribute to the chat, if it would be helpful.
On Jan 14, 2009, at 7:43 AM, Jeff Squyres wrote:
> Ya, that does seem weird to me, but I never fully grokked the whole
> mpool / allocator scheme (I haven't had to interact with that part
> of the code much).
> Would it be useful to get on the phone and discuss this stuff?
> On Jan 14, 2009, at 1:11 AM, Eugene Loh wrote:
>> Thanks for the reply. I kind of understand, but it's rather
>> weird. The BTL calls mca_mpool_base_module_create() to create a
>> pool of memory, but the BTL has no say how big of a pool to
>> create? Could you imagine having a memory allocation routine
>> ("malloc" or something) that didn't allow you to control the size
>> of the allocation? Instead, the allocation routine determines the
>> size. That's weird. I must be missing something about how this is
>> supposed to work.
>> E.g., I see that there is a "resources" argument
>> (mca_mpool_base_resources_t). Maybe that structure should be
>> expanded to include a "size" field?
>> Or, maybe I should bypass mca_mpool_base_module_create()/
>> mca_mpool_sm_init() and just call
>> mca_common_sm_mmap_init() directly, the way mca/coll/sm does
>> things. That would allow me to specify the size of the file.
>> George Bosilca wrote:
>>> The simple answer is you can't. The mpool is loaded before the
>>> BTLs and on Linux the loader use the RTLD_NOW flag (i.e. all
>>> symbols have to be defined or the dlopen call will fail).
>>> Moreover, there is no way in Open MPI to exchange information
>>> between components except a global variable or something in the
>>> mca/common. In other words there is no way for you to call from
>>> the mpool a function from the sm BTL.
>>> On Jan 13, 2009, at 19:22 , Eugene Loh wrote:
>>>> With the sm BTL, there is a file that each process mmaps in for
>>>> shared memory.
>>>> I'm trying to get mpool_sm to size the file appropriately. So,
>>>> I would like mpool_sm to call some mca_btl_sm function that
>>>> provides a good guess of the size. (mpool_sm creates and mmaps
>>>> the file, but the size depends on parameters like eager limit
>>>> and max frag size that are known by the btl_sm.)
>> devel mailing list
> Jeff Squyres
> Cisco Systems
> devel mailing list