On Oct 25, 2010, at 20:22 , Jeff Squyres wrote:
> I dug into this a bit.
> The problem is in the SM BTL init where it's waiting for all of the peers to set seg_inited in shared memory (so that it knows everyone has hit that point). We loop on calling opal_progress while waiting.
> The problem is that opal_progress() is not returning (!).
> It appears that libevent's poll_dispatch() function is somehow getting an infinite timeout -- it *looks* like libevent is determining that there are no timers active, so it decides to set an infinite timeout (i.e., block) when it calls poll(). Specifically, event.c:1524 calls timeout_next(), which sees that there are no timer events active and resets tv_p to NULL. We then call the underlying fd-checking backend with an infinite timeout.
> Anyone more familiar with libevent's internals know why this is happening / if this is a change since the old version?
I did some digging too. My conclusion is somehow [slightly] different.
1. Not all processes deadlock in btl_sm_add_procs. The process that setup the shared memory area, is going forward, and block later in a barrier. Why do we have a barrier in MPI_Init, it's another question, but is not related to the problem at hand here.
2. All other processes, loop around the opal_progress, until they got a message from all other processes. The variable used for counting is somehow updated correctly, but we still call opal_progress. I couldn't figure out is we loop more that we should, or if opal_progress doesn't return. However, both of these possibilities look very unlikely to me: the loop in the sm_add_procs is pretty straightforward, and I couldn't find any loops in opal_progress. I wonder if some of the messages get lost on the exchange.
3. If I unblock the situation by hand, everything goes back to normal. NetPIPE runs to completion but the performances are __really__ bad. On my test machine I get around 2000Mbs, when the expected value is at least 10 times more. Similar finding on the latency side, we're now at 1.65 micro-sec up from the usual 0.35 we had before.
> On Oct 25, 2010, at 6:07 PM, Jeff Squyres wrote:
>> On Oct 25, 2010, at 3:21 PM, George Bosilca wrote:
>>> So now we're in good shape, at least for compiling. IB and TCP seem to work, but SM deadlock.
>> Are you debugging this, or are we? (i.e., me/Ralph)
>> Jeff Squyres
>> For corporate legal information go to:
> Jeff Squyres
> For corporate legal information go to:
> devel mailing list