Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] RE : RE : Implementation of MPI_Iprobe
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2011-09-28 08:29:05


Why not use pre-posted non-blocking receives and MPI_WAIT_ANY?

If you effectively randomize the order of the requests in the array, you can basically control the fairness.

On Sep 27, 2011, at 10:05 PM, Sébastien Boisvert wrote:

> Hello/bonjour,
>
>
> I modified your patch so that in the worse case exactly comm->num_procs are probed (I just changed a < for a <= in the second for loop).
>
> The behavior of your patch was to probe a maximum of comm->num_procs - 1, thus hanging if comm->last_probed has to be probed.
>
>
> With the patch, I get a few good time slices, but most are not productive, see below.
>
>
> Therefore I strongly think that you were right to say that my program should not use MPI_ANY_SOURCE in the first place.
>
>
> I'll stick with my round-robin reception in which many calls to MPI_Iprobe says "no message".
>
>
> In my round-robin reception, I guess that the opal_progress() call in mca_pml_ob1_iprobe (ompi/mca/pml/ob1/pml_ob1_iprobe.c) for all the calls
> to MPI_Iprobe with no match is *really* what helps draining the messages and presumably contributing to the fairness of the whole thing.
>
>
>
> What do you think ?
>
>
>
>
> If I run Ray in gdb:
>
> (gdb) run
> Starting program: /home/sboisver12/git-clones/ray/Ray
> [Thread debugging using libthread_db enabled]
> [New Thread 0x40a00940 (LWP 16867)]
> [New Thread 0x41401940 (LWP 16869)]
>
>
> I see two new threads that are spawned.
>
> The draining of messages is also done in a "progress thread", right ?
>
>
>
> Thank you.
>
>
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.47 s Speed= 90354 Sent= 30 (processMessages: 30, processData: 0) Received= 30 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.57 s Speed= 93748 Sent= 26 (processMessages: 26, processData: 0) Received= 26 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.67 s Speed= 92086 Sent= 66 (processMessages: 66, processData: 0) Received= 66 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.77 s Speed= 93501 Sent= 104 (processMessages: 104, processData: 0) Received= 104 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.87 s Speed= 94512 Sent= 85 (processMessages: 85, processData: 0) Received= 85 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.97 s Speed= 93262 Sent= 138 (processMessages: 138, processData: 0) Received= 138 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.07 s Speed= 94944 Sent= 125 (processMessages: 125, processData: 0) Received= 125 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.17 s Speed= 95377 Sent= 119 (processMessages: 119, processData: 0) Received= 119 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.27 s Speed= 91206 Sent= 129 (processMessages: 129, processData: 0) Received= 129 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.37 s Speed= 92937 Sent= 83 (processMessages: 83, processData: 0) Received= 83 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.47 s Speed= 90803 Sent= 142 (processMessages: 142, processData: 0) Received= 142 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.57 s Speed= 94807 Sent= 87 (processMessages: 87, processData: 0) Received= 87 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.67 s Speed= 92527 Sent= 176 (processMessages: 176, processData: 0) Received= 176 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.77 s Speed= 93514 Sent= 151 (processMessages: 151, processData: 0) Received= 151 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.87 s Speed= 94102 Sent= 81 (processMessages: 81, processData: 0) Received= 81 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.97 s Speed= 95072 Sent= 104 (processMessages: 104, processData: 0) Received= 104 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.07 s Speed= 90934 Sent= 99 (processMessages: 99, processData: 0) Received= 99 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.17 s Speed= 92640 Sent= 117 (processMessages: 117, processData: 0) Received= 117 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.27 s Speed= 93260 Sent= 84 (processMessages: 84, processData: 0) Received= 84 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.37 s Speed= 94774 Sent= 134 (processMessages: 134, processData: 0) Received= 134 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.47 s Speed= 94714 Sent= 84 (processMessages: 84, processData: 0) Received= 84 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.57 s Speed= 94921 Sent= 55 (processMessages: 55, processData: 0) Received= 55 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.67 s Speed= 94461 Sent= 48 (processMessages: 48, processData: 0) Received= 48 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.77 s Speed= 93824 Sent= 58 (processMessages: 58, processData: 0) Received= 58 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.87 s Speed= 90685 Sent= 131 (processMessages: 131, processData: 0) Received= 131 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.97 s Speed= 92930 Sent= 65 (processMessages: 65, processData: 0) Received= 65 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.07 s Speed= 93235 Sent= 38 (processMessages: 38, processData: 0) Received= 38 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.17 s Speed= 92487 Sent= 56 (processMessages: 56, processData: 0) Received= 56 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.27 s Speed= 92559 Sent= 61 (processMessages: 61, processData: 0) Received= 61 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.37 s Speed= 91398 Sent= 47 (processMessages: 47, processData: 0) Received= 47 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.47 s Speed= 94528 Sent= 72 (processMessages: 72, processData: 0) Received= 72 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.57 s Speed= 93548 Sent= 112 (processMessages: 112, processData: 0) Received= 112 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.67 s Speed= 94701 Sent= 54 (processMessages: 54, processData: 0) Received= 54 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.77 s Speed= 94900 Sent= 50 (processMessages: 50, processData: 0) Received= 50 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.87 s Speed= 92868 Sent= 43 (processMessages: 43, processData: 0) Received= 43 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.97 s Speed= 92996 Sent= 170 (processMessages: 54, processData: 116) Received= 170 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.07 s Speed= 88456 Sent= 93 (processMessages: 77, processData: 16) Received= 93 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.17 s Speed= 88194 Sent= 874 (processMessages: 83, processData: 791) Received= 874 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.27 s Speed= 84966 Sent= 439 (processMessages: 79, processData: 360) Received= 439 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.37 s Speed= 83100 Sent= 406 (processMessages: 67, processData: 339) Received= 406 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.47 s Speed= 88243 Sent= 215 (processMessages: 54, processData: 161) Received= 215 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.57 s Speed= 87354 Sent= 550 (processMessages: 201, processData: 349) Received= 550 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.67 s Speed= 94519 Sent= 55 (processMessages: 55, processData: 0) Received= 55 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.77 s Speed= 90221 Sent= 79 (processMessages: 79, processData: 0) Received= 79 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.87 s Speed= 92418 Sent= 146 (processMessages: 146, processData: 0) Received= 146 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.97 s Speed= 94045 Sent= 68 (processMessages: 68, processData: 0) Received= 68 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.07 s Speed= 94432 Sent= 64 (processMessages: 64, processData: 0) Received= 64 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.17 s Speed= 90937 Sent= 68 (processMessages: 68, processData: 0) Received= 68 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.27 s Speed= 88159 Sent= 62 (processMessages: 62, processData: 0) Received= 62 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.37 s Speed= 94526 Sent= 84 (processMessages: 84, processData: 0) Received= 84 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.47 s Speed= 93362 Sent= 73 (processMessages: 73, processData: 0) Received= 73 Balance= 0
> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.57 s Speed= 90062 Sent= 45 (processMessages: 45, processData: 0) Received= 45 Balance= 0
>
>
>> ________________________________________
>> De : devel-bounces_at_[hidden] [devel-bounces_at_[hidden]] de la part de George Bosilca [bosilca_at_[hidden]]
>> Date d'envoi : 27 septembre 2011 16:44
>> À : Open MPI Developers
>> Objet : Re: [OMPI devel] RE : Implementation of MPI_Iprobe
>>
>> Sebastien,
>>
>> Please try the attached patch. It is made against the trunk, so you might have to adapt it slightly.
>>
>> Let me know of the outcome, so we can decide if it's worth pushing it in the next releases.
>>
>> Thanks,
>> george.
>>
>>
>
>
>
> Sébastien<iprobe_starvation_fixed.patch>_______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/