Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] mca_pml_ob1_rdma_btls and leave_pinned logic
From: Roman Cheplyaka (roman.cheplyaka_at_[hidden])
Date: 2009-10-01 11:24:46

Anyone care to explain?

All I want is to see how non-eager RDMA can be used by ob1 if
leave_pinned is FALSE.

2009/9/29 Roman Cheplyaka <roman.cheplyaka_at_[hidden]>:
> Thanks for your explanation, George. However:
> suppose we have leave_pinned = FALSE. Then we go to
> mca_mpool_rdma_find. There we try to find suitable registration in the
> cache. Suppose we cannot (cache's empty). Then NULL registration is
> returned, BTL is skipped and 0 is returned from mca_pml_ob1_rdma_btls.
> This way RDMA never happends.
> (To make it clear: this is not some theoretical situation, this is
> what I'm actually observing. OpenMPI 1.2.9) Can somebody explain
> please?
> 2009/9/29 George Bosilca <bosilca_at_[hidden]>:
>> Roman,
>> Before going into explaining the logic, let me state that the memory is
>> registered (if required/supported) by the BTLs. However, this is done only
>> at the moment when the memory segment is involved in any kind of
>> communication.
>> We do not want to replicate this at the PML level, in order to make sure
>> that the amount of memory registered at any moment is minimal. In other
>> words, the PML leave the decision on when to register and when to unregister
>> to the BTLs. However, in order to speedup the code a little bit (and to keep
>> things tidy), the PML will help the BTLs to work around the memory
>> registration issue. And the code you pointed out is exactly the place where
>> we do it.
>> We need to register the memory if leave_pinned is TRUE, as registering will
>> leave a trace. If leave_pinned is FALSE then we only check if somehow this
>> memory is not already registered (by some BTL). In this case, there is no
>> need to create a registration in the PML (if required the BTL will do it
>> when needed).
>>  george.
>> On Sep 28, 2009, at 13:44 , Roman Cheplyaka wrote:
>>> Hi,
>>> I'm trying to dig into OpenMPI sources but have some problems. Can
>>> anyone explain the logic of the following code from
>>> mca/pml/ob1/pml_ob1_rdma.c please?
>>>           if(!mca_pml_ob1.leave_pinned) {
>>>               /* look through existing registrations */
>>>               btl_mpool->mpool_find(btl_mpool, base, size, &reg);
>>>           } else {
>>>               /* register the memory */
>>>               btl_mpool->mpool_register(btl_mpool, base, size, 0, &reg);
>>>           }
>>> It seems to me that we should register new memory if leave_pinned is
>>> FALSE (i.e. no existing registrations avaliable) and search through
>>> existing registrations otherwise, but the logic is inversed here.

Roman I. Cheplyaka