Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] self-messaging questions
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2010-10-15 06:21:18


On Oct 14, 2010, at 7:50 PM, Gus Correa wrote:

> *1) Can a process send a message to itself?*
>
> The book "MPI: The complete reference, Vol. 1, 2nd Ed.", by Snir et al.,
> page 42, section 2.9.9 (Comments on Receive) says:
>
> "Therefore, it is unsafe and non-portable to send self-messages with the standard mode, blocking send and receive operations described so far,
> since this may lead to deadlock."

Correct.

> *2) Would it be safe and portable to send messages to self, if one uses non-blocking send and recv (MPI_Isend, MPI_Irecv), or with persistent
> communication (MPI_Init_[Send/Recv], MPI_Startall, MPI_Waitall) ?*

Yes.

> On the other hand, the OpenMPI FAQ seems to say something different,
> i.e., that self-messaging is OK, as long as the
> "self" BTL is turned on:
>
> http://www.open-mpi.org/faq/?category=all#tcp-btl

Also correct.

We made a software engineering choice with the BTLs (i.e., the OB1-based transports) that they would not have the ability to send to themselves. Instead, we created a "self" BTL that has all of the logic for sending messages to myself. In this way, we didn't have to replicate all of that logic in every other BTL.

So it's not really a performance optimization -- it's really a software engineering issue (i.e., avoid code duplication).

> *3) Is this a particular feature or extension provided by OpenMPI,
> beyond the MPI standard?*

Nope. Sending to self is something that the MPI standard implicitly guarantees must work.

> *4) If I write a program with self-messaging, will it
> be portable and run safely when compiled with other MPI implementations?*

Yes. Provided you use non-blocking communications, per #2.

> Of course, I can use an "if" conditional to copy the data to
> the output buffer, instead of sending the message to self,
> although this makes the code somewhat ugly.
> This brings up yet another question:
>
> *5) Which one is better/faster: sending messages to self,
> or copying the data to the output buffer?*

If the copy is a straight memcpy, it's probably marginally faster to use memcpy only because you're not making as many function calls to get down into the MPI progression engine. We didn't really take any pains to highly optimize the send-to-self case, but it should perform well.

You might want to benchmark memcpy vs. MPI_Sendrecv() to self (for example) and see if there's a noticeable difference.

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/