Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] RFC: eliminating "descriptor" argument from sendi function
From: Eugene Loh (Eugene.Loh_at_[hidden])
Date: 2009-02-23 18:07:30


Eugene Loh wrote:

> Actually, there may be a more important issue here.
>
> Currently, the PML chooses the BTL first. Once the BTL choice is
> established, only then does the PML choose between sendi and send.
>
> Currently, it's also the case that we're spending a lot of time in the
> PML doing a bunch of stuff that's totally unnecessary if the sendi
> succeeds. So, we're neutralizing much of the advantage sendi is
> supposed to provide.
>
> So, I'm changing the PML to invoke sendi much sooner. The way I'm
> doing this is to loop over BTLs, looking for a sendi that exists and
> succeeds. If I find one, I'm done. If I don't, I have to go with the
> standard send code path.
>
> The logic, as I just described it, allows that multiple sendi
> functions could fail and that the send that is ultimately used might
> be for a different BTL than for any of the failing sendi's. This
> would suggest that I do NOT want failing sendi's leaving any side
> effects (like allocated descriptors).
>
> Is my proposed logic bad? Should I implement things another way?
> E.g., if I find a sendi function, use that BTL even if the sendi
> failed and another BTL might have a sendi that could succeed? Or,
> does my proposed change provide the justification for my pulling
> descriptor allocations out of the sendi functions?

Here's another way of looking at it.

The current PML send code does this:

    set_up_expensive_send_request(&sendreq);
    for ( btl = ... ) {
        if ( SUCCESS == sendi() ) return SUCCESS;
        if ( SUCCESS == send(&sendreq) ) return SUCESS;
    }

That is, we try one BTL after another. For each one, we try sendi
first. So, each sendi() that fails is immediately followed by a send()
of the same BTL. It's okay for a sendi() to do prep work for the send()
of the same BTL. This scheme does a bunch of expensive send-request
initialization that is unnecessary if the sendi(), which doesn't need
the send request, succeeds.

My proposed PML send logic is this:

    for ( btl = ... ) {
        if ( SUCCESS == sendi() ) return SUCCESS;
    }
    set_up_expensive_send_request(&sendreq);
    for ( btl = ... ) {
        if ( SUCCESS == send(&sendreq) ) return SUCCESS;
    }

That is, if I can find a sendi() function, I use it. Only if I can't
find any sendi() do I set up the send request and call send() functions.

This is why I would like sendi() functions to have no side effects...
e.g., no allocated descriptors.