Jeff Squyres wrote:
> On Dec 12, 2008, at 11:46 AM, Eugene Loh wrote:
>> FWIW, I've run into the need for this a few times due to HPCC tests
>> on large (>100 MPI procs) nodes or multicore systems. HPCC (among
>> other things) looks at the performance of a single process while all
>> other np-1 processes spinwait -- or of a single pingpong pair while
>> all other np-2 processes wait. I'm not 100% sure what's going on,
>> but I'm guessing that the hard spinning of waiting processes hits
>> the memory system or some other resource, degrading the performance
>> of working processes. This is on nodes that are not oversubscribed.
> I guess I could <waving hands> see how shmem kinds of communication
> could lead to this kind of bottleneck, and that increasing core
> counts would magnify the effect. It would be good to understand if
> shmem activity is the cause of the slowdown to know if this is a
> good rationale datapoint for whether we should do blocking progress
> (or, more specifically, whether we need to increase the priority of
> implementing blocking progress).
I don't understand all of what's going on here, but I/we've seen this
sort of "catastrophic degradation" on two large (>100 processes) nodes
of rather different architectures. Prototypes seem to indicate that
blocking *or* directed polling seems to address the problem, but those
are preliminary findings that are not backed up by sound understanding
of what's going on under the hood. Yes, still handwaving.