Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] RFC: sm Latency
From: Eugene Loh (Eugene.Loh_at_[hidden])
Date: 2009-01-20 20:04:11

Patrick Geoffray wrote:

>Eugene Loh wrote:
>>>replace the fifo¡¦s with a single link list per process in shared
>>>memory, with senders to this process adding match envelopes
>>>atomically, with each process reading its own link list (multiple
>>*) Doesn't strike me as a "simple" change.
>Actually, it's much simpler than trying to optimize/scale the N^2
>implementation, IMHO.
1) The version I talk about is already done. Check my putbacks. "Already
done" is easier! :^)

2) The two ideas are largely orthogonal. The RFC talks about a variety
of things: cleaning up the sendi function, moving the sendi call up
higher in the PML, bypassing the PML receive-request structure (similar
to sendi), and stream-lining the data convertors in common cases. Only
one part of the RFC (directed polling) overlaps with having a single
FIFO per receiver.

>>*) Not sure this addresses all-to-all well. E.g., let's say you post a
>>receive for a particular source. Do you then wade through a long FIFO
>>to look for your match?
>The tradeoff is between demultiplexing by the sender, which cost in time
>and in space, or by the receiver, which cost an atomic inc. ANY_TAG
>forces you to demultiplex on the receive side anyway. Regarding
>all-to-all, it won't be more expensive if the receives are pre-posted,
>and they should be.
Not sure I understand this paragraph. I do, however, think there are
great benefits to the single-receiver-queue model. It implies congestion
on the receiver side in the many-to-one case, but if a single receiver
is reading all those messages anyhow, message-processing is already
going to throttle the message rate. The extra "bottleneck" at the FIFO
might never be seen.

>>What the RFC talks about is not the last SM development we'll ever
>>need. It's only supposed to be one step forward from where we are
>>today. The "single queue per receiver" approach has many advantages,
>>but I think it's a different topic.
>But is this intermediate step worth it or should we (well, you :-) ) go
>directly for the single queue model ?
To recap:
1) The work is already done.
2) The single-queue model addresses only one of the RFC's issues.
3) I'm a fan of the single-queue model, but it's just a separate discussion.