Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] [EXTERNAL] Re: Configuration for rendezvous and eager protocols: two-sided comm
From: Siddhartha Jana (siddharthajana24_at_[hidden])
Date: 2014-01-13 16:42:29


Good evening

A quick question,
If I set the short message size limit to an unreasonably large number as
below, will this request be honored by OpenMPI

mpirun --mca btl tcp,self \
           --mca btl_tcp_eager_limit 2097152 \
           --mca btl_tcp_rndv_eager_limit 2097152 \
           --mca btl_tcp_max_send_size 2097152 \
           -np 2 ./a.out

Just wanted to confirm whether OpenMPI has some strict limits when it comes
to detecting whether a message size should be treated as short or whether
the user has the final say.

Thanks,
Sid

On 27 December 2013 03:01, Siddhartha Jana <siddharthajana24_at_[hidden]>wrote:

> Thanks a lot Pasha! The difference between btl_*_rndv_eager_limit and btl_*_eager_limit
> is now clear.
>
> -- Sid
>
>
> On 26 December 2013 10:19, Shamis, Pavel <shamisp_at_[hidden]> wrote:
>
>> Hey Sid ;)
>>
>> Please see inline.
>>
>>
>>
>> 1.1. What is the meaning of "phase 1" fragment?
>>
>> *[Shamis, Pavel] Phase one of the rendezvous (RNDV) protocol.
>> Essentially, the first request of the RNDV we send as an eager message of
>> size N,*
>>
>> *Receiver unpacks the messages and switches RNDV get or put semantics.*
>>
>> 1.2. Is my understanding correct that the btl_*_eager_limit is
>> applicable only in case of one-sided communication?
>>
>>
>>
>> *[Shamis, Pavel] It is actually used for small messages in the
>> send-receive semantics. The idea is that the cost of memory registration is
>> relatively high,*
>>
>> *To avoid it, we memcopy the data to a pre-registred /allocated buffer.*
>>
>> *Best,*
>>
>> *P.*
>>
>>
>>
>> On 16 December 2013 14:36, Jeff Squyres (jsquyres) <jsquyres_at_[hidden]>
>> wrote:
>>
>> Everything that Brian said, plus: note that the MCA param that Christoph
>> mentioned is specifically for the "sm" (shared memory) transport. Each
>> transport has their own set of MCA params (e.g., mca_btl_tcp_eager_limit,
>> and friends).
>>
>>
>>
>> On Dec 16, 2013, at 3:19 PM, "Barrett, Brian W" <bwbarre_at_[hidden]>
>> wrote:
>>
>> > Siddhartha -
>> >
>> > Christoph mentioned how to change the cross-over for shared memory, but
>> it's really per-transport (so you'd have to change it for your off-node
>> transport as well). That's all in the FAQ you mentioned, so hopefully you
>> can take it from there. Note that, in general, moving the eager limits has
>> some unintended side effects. For example, it can cause more / less
>> copies. It can also greatly increase memory usage.
>> >
>> > Good luck,
>> >
>> > Brian
>> >
>> > On 12/16/13 1:49 AM, "Siddhartha Jana" <siddharthajana24_at_[hidden]>
>> wrote:
>> >
>> >> Thanks Christoph.
>> >> I should have looked into the FAQ section on MCA params setting @ :
>> >> http://www.open-mpi.org/faq/?category=tuning#available-mca-params
>> >>
>> >> Thanks again,
>> >> -- Siddhartha
>> >>
>> >>
>> >> On 16 December 2013 02:41, Christoph Niethammer <niethammer_at_[hidden]>
>> wrote:
>> >>> Hi Siddhartha,
>> >>>
>> >>> MPI_Send/Recv in Open MPI implements both protocols and chooses based
>> on the message size which one to use.
>> >>> You can use the mca parameter "btl_sm_eager_limit" to modify the
>> behaviour.
>> >>>
>> >>> Here the corresponding info obtained from the ompi_info tool:
>> >>>
>> >>> "btl_sm_eager_limit" (current value: <4096>, data source: default
>> value)
>> >>> Maximum size (in bytes) of "short" messages (must be >= 1)
>> >>>
>> >>> Regards
>> >>> Christoph Niethammer
>> >>>
>> >>> --
>> >>>
>> >>> Christoph Niethammer
>> >>> High Performance Computing Center Stuttgart (HLRS)
>> >>> Nobelstrasse 19
>> >>> 70569 Stuttgart
>> >>>
>> >>> Tel: ++49(0)711-685-87203
>> >>> email: niethammer_at_[hidden]
>> >>> http://www.hlrs.de/people/niethammer
>> >>>
>> >>>
>> >>>
>> >>> ----- Ursprüngliche Mail -----
>> >>> Von: "Siddhartha Jana" <siddharthajana24_at_[hidden]>
>> >>> An: "OpenMPI users mailing list" <users_at_[hidden]>
>> >>> Gesendet: Samstag, 14. Dezember 2013 13:44:12
>> >>> Betreff: [OMPI users] Configuration for rendezvous and eager
>> protocols: two-sided comm
>> >>>
>> >>>
>> >>>
>> >>> Hi
>> >>>
>> >>>
>> >>> In OpenMPI, are MPI_Send, MPI_Recv (and friends) implemented using
>> rendezvous protocol or eager protocol?
>> >>>
>> >>>
>> >>> If both, is there a way to choose one or the other during runtime or
>> while building the library?
>> >>>
>> >>>
>> >>> If there is a threshold of the message size that dictates the
>> protocol to be used, is there a way I can alter that threshold value?
>> >>>
>> >>>
>> >>> If different protocols were used for different versions of the
>> library in the past, could someone please direct me to the exact version
>> numbers of the implementations that used one or the other protocol?
>> >>>
>> >>>
>> >>> Thanks a lot,
>> >>> Siddhartha
>> >>> _______________________________________________
>> >>> users mailing list
>> >>> users_at_[hidden]
>> >>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> >>> _______________________________________________
>> >>> users mailing list
>> >>> users_at_[hidden]
>> >>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> >>
>> >
>> >
>> > --
>> > Brian W. Barrett
>> > Scalable System Software Group
>> > Sandia National Laboratories
>> > _______________________________________________
>> > users mailing list
>> > users_at_[hidden]
>> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>> --
>>
>> Jeff Squyres
>> jsquyres_at_[hidden]
>> For corporate legal information go to:
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
>