Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] trouble using --mca mpi_yield_when_idle 1
From: Eugene Loh (Eugene.Loh_at_[hidden])
Date: 2008-12-12 17:17:25


Jeff Squyres wrote:

> On Dec 12, 2008, at 11:46 AM, Eugene Loh wrote:
>
>> FWIW, I've run into the need for this a few times due to HPCC tests
>> on large (>100 MPI procs) nodes or multicore systems. HPCC (among
>> other things) looks at the performance of a single process while all
>> other np-1 processes spinwait -- or of a single pingpong pair while
>> all other np-2 processes wait. I'm not 100% sure what's going on,
>> but I'm guessing that the hard spinning of waiting processes hits
>> the memory system or some other resource, degrading the performance
>> of working processes. This is on nodes that are not oversubscribed.
>
> I guess I could <waving hands> see how shmem kinds of communication
> could lead to this kind of bottleneck, and that increasing core
> counts would magnify the effect. It would be good to understand if
> shmem activity is the cause of the slowdown to know if this is a
> good rationale datapoint for whether we should do blocking progress
> (or, more specifically, whether we need to increase the priority of
> implementing blocking progress).

I don't understand all of what's going on here, but I/we've seen this
sort of "catastrophic degradation" on two large (>100 processes) nodes
of rather different architectures. Prototypes seem to indicate that
blocking *or* directed polling seems to address the problem, but those
are preliminary findings that are not backed up by sound understanding
of what's going on under the hood. Yes, still handwaving.