Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] MPI_AllReduce() deadlock on IB
From: Brock Palen (brockp_at_[hidden])
Date: 2011-03-16 11:27:06


I have a user whos code when ran on ethernet performs fine. When ran on verbs based IB the code deadlocks in an MPI_AllReduce() call.

We are using openmpi/1.4.3 with the intel compilers.

I poked at the running code with padb and I get the following:

0....5....1....5....2....5....3....5....4....5....
,,---,-,-,----,--,--,,-,RRRRRRRR,---,----,,--,-,-,
,,-,-,,,-,,--,-,,-,-,-,-RRRRRRRR-,-,---,,,--,,---,
,,---,-,,,,-,-,,-,-,----RRRRRRRR,----,-,--,,-----,
--,,-,-,,,,-,,------,,--RRRRRRRR,,----,,--,------,

For multiple runs which ranks are stuck in AllReduce() changes,
Is there any open bugs? I found one but only on shared memory and our version should be new enough (from what I could tell) to avoid it.

Thanks, what should I look for to diagnose the issue?

Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp_at_[hidden]
(734)936-1985