Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] IMB-MPI broadcast test stalls for large core counts: debug ideas?
From: Rahul Nabar (rpnabar_at_[hidden])
Date: 2010-08-24 13:34:35


On Mon, Aug 23, 2010 at 6:39 PM, Richard Treumann <treumann_at_[hidden]> wrote:
> It is hard to imagine how a total data load of 41,943,040 bytes could be a
> problem. That is really not much data. By the time the BCAST is done, each
> task (except root) will have received a single half meg message form one
> sender. That is not much.

Thanks very much for your comments Dick! I'm somewhat new to MPI so
appreciate all the advice I can get.My main roadblock is I'm not sure
how to attack this problem more? How can I obtain more diagnostic
output to help me trace what the origin of this "broadcast stall" is?
So far I've obtained a stack trace via padb (
http://dl.dropbox.com/u/118481/padb.log.new.new.txt ) but that is
about all.

Any suggestions as to what else I could try? Would a full dump by
something like tcpdump or wireshark on the packets passing the network
be of any relevance? Or is there something useful to be known from the
switch side? The technology is fairly new for HPC (Chelsio 10GigE
adapters + Cisco Nexus5000 switches). So I wouldn't rule out some
strange hardware or firmware bug that's tickled by this particular
suite of tests. I'm grasping at straws here.

 [ On the other hand I'm fairly new so I wouldn't rule out some silly
setting by me as well. ]

-- 
Rahul