Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] RDMA pipeline
From: George Bosilca (bosilca_at_[hidden])
Date: 2008-02-19 22:40:46

Actually, it restores the original behavior. The RDMA operations were
pipelined before the r15247 commit, independent of the fact that they
had mpool or not. We were actively using this behavior in the message
logging framework to hide the cost of the local storage of the
payload, and we were quite surprised when we realized that it

If a BTL don't want to use pipeline for RDMA operations, it can set
the RDMA fragment size to the max value, and this will automatically
disable the pipeline. However, if the BTL support pipeline with the
trunk version today it is not possible to activate it. Moreover, in
the current version the parameters that define the BTL behavior are
blatantly ignored, as the PML make high level assumption about what
they want to do.


On Feb 19, 2008, at 3:03 PM, Gleb Natapov wrote:

> On Tue, Feb 19, 2008 at 02:13:30PM -0500, George Bosilca wrote:
>> Few days ago during some testing I realize that the RDMA pipeline was
>> disabled for MX and Elan (I didn't check for the others). A quick
>> look
>> into the source code, pinpointed the problem into the pml_ob1_rdma.c
>> file, and it seems that the problem was introduced by commit 15247.
>> The
>> problem comes from the usage of the dummy registration, which is
>> set for
>> all non mpool friendly BTL. Later on this is checked against NULL
>> (and of
>> course it fails), which basically disable the RDMA pipeline.
> Do you mean that mca_pml_ob1_send_request_start_rdma() is used for
> rendezvous sends? I will be very surprised if ompi 1.2 works
> differently. It assumes that if btl has no mpool then entire message
> buffer
> is registered and no pipeline is needed. Trunk does the same but
> differently. OpenIB also choose this route if buffer memory is
> allocated
> by MPI_alloc_mem().
>> I'll enable the RDMA pipeline back in 2 days if I don't hear anything
>> back. Attached is the patch that fix this problem.
> I am not sure why you need pipeline for BTLs that don't require
> registration, but by applying this patch you'll change how ompi
> behaves
> from v1.0. (unless I miss something, then please provide more
> explanations).
> --
> Gleb.
> _______________________________________________
> devel mailing list
> devel_at_[hidden]

  • application/pkcs7-signature attachment: smime.p7s