Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] mca_btl_sm_sendi question
From: George Bosilca (bosilca_at_[hidden])
Date: 2009-02-25 14:04:52


On Feb 25, 2009, at 10:36 , Eugene Loh wrote:

> George Bosilca wrote:
>
>> On Feb 24, 2009, at 18:08 , Eugene Loh wrote:
>>
>>> (Probably this message only for George, but I'll toss it out to
>>> the alias/archive.)
>>
> Actually, maybe Rich should weigh in here, too. This relates to the
> overflow mechanism in MCA_BTL_SM_FIFO_WRITE.
>
>>> I have a question about the sm sendi() function. What should
>>> happen if the sendi() function attempts to write to the FIFO, but
>>> the FIFO is full?
>>
>> The write should not be queued except in the case where the whole
>> data referred by the convertor is copied out of the user memory.
>
> And this is indeed the case. The data-convertor copy completed
> successfully.
>
>> If the FIFO is full, the best will be to allocate the descriptor
>> and give it back to the PML.
>
> Why? The data has been copied out of the user's buffer. The
> pointer to that data has been queued for sending. (It hasn't been
> queued in the FIFO, which is full, but it has been queued in the
> pending-send list.)

As I previously state, if the data is copied out of the user buffer,
the sendi should always return success. However, having a queue in the
BTL only duplicates the queue from the PML.

> The FIFO has an overflow mechanism. Actually, prior to my recent
> putbacks, it had two overflow mechanisms. One was to grow the FIFO,
> and the other was to use the pending-send queue. While adding
> support for multiple senders per FIFO and at Rich's suggestion, I
> pulled out the ability to grow the FIFO. (Some number of folks
> didn't even believe that the FIFO-grow stuff even existed or was
> enabled or worked properly.) That still leaves the pending sends.
> So, the "out of resource" return code from the FIFO write is kind of
> spurious. The FIFO write is returning that code even though it has
> accepted the write and queued it up.
>
>>> Currently, it appears that the sendi() function returns an error
>>> code to the PML, which assumes that the sendi() tried to send the
>>> message but failed and so just tried to allocate a descriptor.
>>
>> Yes, this is the expected behavior.
>>
>>> But is that what should happen? The condition of the FIFO being
>>> full is a little misleading since the write is still queued for
>>> further progress -- not in the FIFO itself but in the pending-
>>> send queue. This distinction should perhaps not matter to the
>>> upper layers. The upper layers should still view the send as
>>> "completed" (buffered by the MPI implementation to be progressed
>>> later). I would think that the sendi() function should return a
>>> SUCCESS code.
>>
>> If the write is queued then this is more or less a bug. We will
>> nicely cope with this case, because we have this sequence number
>> and we will drop a message duplicate, but we will end-up sending
>> the same message twice. The problem is that I don't know which of
>> the copies will be used on the receiver side, I guess the first
>> one reaching the receiver.
>
> Arrgh! When the primary mechanism (FIFO) starts getting congested,
> we start pumping duplicate messages into the system?

If the BTL queue the send internally and returns an error, then the
PML will:
- go back in the mca_pml_ob1_send_request_start with the error set to
OUT_OF_RESSOURCES
- will continue over the list of available BTL for the eager and try
to send the same message again.
- in the case no more BTLs are available it will add the request to
the pending queue, and it will reschedule it later.

So the answer is yes, if a BTL returns an error while adding the data
in its own queues, then we will duplicate the send operation.

   george.

> The proper fix (IMHO) is to have the sendi function return a SUCCESS
> code once it's written the message and the pointer to the message.
> And, once it's written those two things, it seems to me to be a bug
> to return any other code.
>
>>> Relevent source code is
>>>
>>> PML, line 496
>>> https://svn.open-mpi.org/source/xref/ompi_1.3/ompi/mca/pml/ob1/pml_ob1_sendreq.c
>>> #496
>>>
>>> BTL, line 785
>>> https://svn.open-mpi.org/source/xref/ompi_1.3/ompi/mca/btl/sm/
>>> btl_sm.c#785
>>>
>>> FIFO write, line 18
>>> https://svn.open-mpi.org/opengrok/xref/ompi_1.3/ompi/mca/btl/sm/btl_sm_fifo.h
>>> #18
>>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel