Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI devel] mca_pml_ob1_rdma_btls and leave_pinned logic
From: Roman Cheplyaka (roman.cheplyaka_at_[hidden])
Date: 2009-09-29 13:36:36


Thanks for your explanation, George. However:
suppose we have leave_pinned = FALSE. Then we go to
mca_mpool_rdma_find. There we try to find suitable registration in the
cache. Suppose we cannot (cache's empty). Then NULL registration is
returned, BTL is skipped and 0 is returned from mca_pml_ob1_rdma_btls.
This way RDMA never happends.

(To make it clear: this is not some theoretical situation, this is
what I'm actually observing. OpenMPI 1.2.9) Can somebody explain
please?

2009/9/29 George Bosilca <bosilca_at_[hidden]>:
> Roman,
>
> Before going into explaining the logic, let me state that the memory is
> registered (if required/supported) by the BTLs. However, this is done only
> at the moment when the memory segment is involved in any kind of
> communication.
>
> We do not want to replicate this at the PML level, in order to make sure
> that the amount of memory registered at any moment is minimal. In other
> words, the PML leave the decision on when to register and when to unregister
> to the BTLs. However, in order to speedup the code a little bit (and to keep
> things tidy), the PML will help the BTLs to work around the memory
> registration issue. And the code you pointed out is exactly the place where
> we do it.
>
> We need to register the memory if leave_pinned is TRUE, as registering will
> leave a trace. If leave_pinned is FALSE then we only check if somehow this
> memory is not already registered (by some BTL). In this case, there is no
> need to create a registration in the PML (if required the BTL will do it
> when needed).
>
>  george.
>
> On Sep 28, 2009, at 13:44 , Roman Cheplyaka wrote:
>
>> Hi,
>> I'm trying to dig into OpenMPI sources but have some problems. Can
>> anyone explain the logic of the following code from
>> mca/pml/ob1/pml_ob1_rdma.c please?
>>
>>           if(!mca_pml_ob1.leave_pinned) {
>>               /* look through existing registrations */
>>               btl_mpool->mpool_find(btl_mpool, base, size, &reg);
>>           } else {
>>               /* register the memory */
>>               btl_mpool->mpool_register(btl_mpool, base, size, 0, &reg);
>>           }
>>
>> It seems to me that we should register new memory if leave_pinned is
>> FALSE (i.e. no existing registrations avaliable) and search through
>> existing registrations otherwise, but the logic is inversed here.
>>
>> --
>> Roman I. Cheplyaka
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>

-- 
Roman I. Cheplyaka