Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] IMB-MPI broadcast test stalls for large core counts: debug ideas?
From: Rahul Nabar (rpnabar_at_[hidden])
Date: 2010-09-01 16:43:17


On Wed, Aug 25, 2010 at 12:14 PM, Jeff Squyres <jsquyres_at_[hidden]> wrote:
> It would simplify testing if you could get all the eth0's to be of one type and on the same subnet, and the same for eth1.
>
> Once you do that, try using just one of the networks by telling OMPI to use only one of the devices, something like this:
>
>    mpirun --mca btl_tcp_if_include eth0 ...

Thanks for all the suggestions guys! We finally got this figured out.
It was the result of two different (hardware specific) bugs in the
RDMA driver. The 10GigE card was advertising a wrong size for the CQ
stack (as far as I understand!).

In case anyone wants to know more, the bugfixes are posted here:

http://www.mail-archive.com/linux-rdma@vger.kernel.org/msg05451.html
http://www.mail-archive.com/linux-rdma@vger.kernel.org/msg05246.html

Cheers!

-- 
Rahul