Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] mca_pml_ob1_rdma_btls and leave_pinned logic
From: Roman Cheplyaka (roman.cheplyaka_at_[hidden])
Date: 2009-09-29 13:36:36


Thanks for your explanation, George. However:
suppose we have leave_pinned = FALSE. Then we go to
mca_mpool_rdma_find. There we try to find suitable registration in the
cache. Suppose we cannot (cache's empty). Then NULL registration is
returned, BTL is skipped and 0 is returned from mca_pml_ob1_rdma_btls.
This way RDMA never happends.

(To make it clear: this is not some theoretical situation, this is
what I'm actually observing. OpenMPI 1.2.9) Can somebody explain
please?

2009/9/29 George Bosilca <bosilca_at_[hidden]>:
> Roman,
>
> Before going into explaining the logic, let me state that the memory is
> registered (if required/supported) by the BTLs. However, this is done only
> at the moment when the memory segment is involved in any kind of
> communication.
>
> We do not want to replicate this at the PML level, in order to make sure
> that the amount of memory registered at any moment is minimal. In other
> words, the PML leave the decision on when to register and when to unregister
> to the BTLs. However, in order to speedup the code a little bit (and to keep
> things tidy), the PML will help the BTLs to work around the memory
> registration issue. And the code you pointed out is exactly the place where
> we do it.
>
> We need to register the memory if leave_pinned is TRUE, as registering will
> leave a trace. If leave_pinned is FALSE then we only check if somehow this
> memory is not already registered (by some BTL). In this case, there is no
> need to create a registration in the PML (if required the BTL will do it
> when needed).
>
>  george.
>
> On Sep 28, 2009, at 13:44 , Roman Cheplyaka wrote:
>
>> Hi,
>> I'm trying to dig into OpenMPI sources but have some problems. Can
>> anyone explain the logic of the following code from
>> mca/pml/ob1/pml_ob1_rdma.c please?
>>
>>           if(!mca_pml_ob1.leave_pinned) {
>>               /* look through existing registrations */
>>               btl_mpool->mpool_find(btl_mpool, base, size, &reg);
>>           } else {
>>               /* register the memory */
>>               btl_mpool->mpool_register(btl_mpool, base, size, 0, &reg);
>>           }
>>
>> It seems to me that we should register new memory if leave_pinned is
>> FALSE (i.e. no existing registrations avaliable) and search through
>> existing registrations otherwise, but the logic is inversed here.
>>
>> --
>> Roman I. Cheplyaka
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>

-- 
Roman I. Cheplyaka