Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Simple program (103 lines) makes Open-1.4.3 hang
From: Jeff Squyres (jsquyres) (jsquyres_at_[hidden])
Date: 2010-11-23 15:17:44


Sorry for the delay in replying - many of us were at SC last week.

Admittedly, I'm looking at your code on a PDA, so I might be missing some things. But I have 2 q's:

1 your send routine doesn't seem to protect from sending to yourself. Correct?

2 you're not using nonblocking sends, which, if I understand your code right, can lead to deadlock. Right? Eg proc A sends to proc b and blocks until b receives. But b is blocking waiting for it's send completion, etc.

I think with your random destinations (which may even be yourself, in which case the blocking send will never complete because you didn't prepost a nomblocking receive) and blocking sends, you can end up with deadlock.

Sent from my PDA. No type good.

On Nov 16, 2010, at 5:21 PM, Sébastien Boisvert <Sebastien.Boisvert.3_at_[hidden]> wrote:

> Dear awesome community,
>
>
> Over the last months, I closely followed the evolution of bug 2043,
> entitled 'sm BTL hang with GCC 4.4.x'.
>
> https://svn.open-mpi.org/trac/ompi/ticket/2043
>
> The reason is that I am developping an MPI-based software, and I use
> Open-MPI as it is the only implementation I am aware of that send
> messages eagerly (powerful feature, that is).
>
> http://denovoassembler.sourceforge.net/
>
> I believe that this very pesky bug remains in Open-MPI 1.4.3, and
> enclosed to this communication are scientific proofs of my claim, or at
> least I think they are ;).
>
>
> Each byte transfer layer has its default limit to send eagerly a
> message. With shared memory (sm), the value is 4096 bytes. At least it
> is according to ompi_info.
>
>
> To verify this limit, I implemented a very simple test. The source code
> is test4096.cpp, which basically just send a single message of 4096
> bytes from a rank to another (rank 1 to 0).
>
> The test was conclusive: the limit is 4096 bytes (see
> mpirun-np-2-Simple.txt).
>
>
>
> Then, I implemented a simple program (103 lines) that makes Open-MPI
> 1.4.3 hang. The code is in make-it-hang.cpp. At each iteration, each
> rank send a message to a randomly-selected destination. A rank polls for
> new messages with MPI_Iprobe. Each rank prints the current time at each
> second during 30 seconds. Using this simple code, I ran 4 test cases,
> each with a different outcome (use the Makefile if you want to reproduce
> the bug).
>
> Before I describe these cases, I will describe the testing hardware.
>
> I use a computer with 32 x86_64 cores (see cat-proc-cpuinfo.txt.gz).
> The computer has 128 GB of physical memory (see
> cat-proc-meminfo.txt.gz).
> It runs Fedora Core 11 with Linux 2.6.30.10-105.2.23.fc11.x86_64 (see
> dmesg.txt.gz & uname.txt).
> Default kernel parameters are utilized at runtime (see
> sudo-sysctl-a.txt.gz).
>
> The C++ compiler is g++ (GCC) 4.4.1 20090725 (Red Hat 4.4.1-2) (see g
> ++--version.txt).
>
>
> I compiled Open-MPI 1.4.3 myself (see config.out.gz, make.out.gz,
> make-install.out.gz).
> Finally, I use Open-MPI 1.4.3 with defaults (see ompi_info.txt.gz).
>
>
>
>
> Now I can describe the cases.
>
>
> Case 1: 30 MPI ranks, message size is 4096 bytes
>
> File: mpirun-np-30-Program-4096.txt
> Outcome: It hangs -- I killed the poor thing after 30 seconds or so.
>
>
>
>
> Case 2: 30 MPI ranks, message size is 1 byte
>
> File: mpirun-np-30-Program-1.txt.gz
> Outcome: It runs just fine.
>
>
>
>
> Case 3: 2 MPI ranks, message size is 4096 bytes
>
> File: mpirun-np-2-Program-4096.txt
> Outcome: It hangs -- I killed the poor thing after 30 seconds or so.
>
>
>
>
> Case 4: 30 MPI ranks, message size if 4096 bytes, shared memory is
> disabled
>
> File: mpirun-mca-btl-^sm-np-30-Program-4096.txt.gz
> Outcome: It runs just fine.
>
>
>
>
>
> A backtrace of the processes in Case 1 is in gdb-bt.txt.gz.
>
>
>
>
> Thank you !
>
> <test4096.cpp>
> <mpirun-np-2-Simple.txt>
> <make-it-hang.cpp>
> <Makefile>
> <cat-proc-cpuinfo.txt.gz>
> <cat-proc-meminfo.txt.gz>
> <dmesg.txt.gz>
> <uname.txt>
> <sudo-sysctl-a.txt.gz>
> <g++--version.txt>
> <config.out.gz>
> <make.out.gz>
> <make-install.out.gz>
> <ompi_info.txt.gz>
> <mpirun-np-30-Program-4096.txt>
> <mpirun-np-30-Program-1.txt.gz>
> <mpirun-np-2-Program-4096.txt>
> <mpirun-mca-btl-^sm-np-30-Program-4096.txt.gz>
> <gdb-bt.txt.gz>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel