Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Brian Barrett (brbarret_at_[hidden])
Date: 2006-01-14 14:10:48


On Jan 13, 2006, at 10:41 PM, Glenn Morris wrote:

> The combination OpenMP + OpenMPI works fine if I restrict the
> application to only 1 OpenMP thread per MPI process (in other words
> the code at least compiles and runs fine with both options on, in this
> limited sense). If I try to use my desired value of 4 OpenMP threads,
> it crashes. It works fine, however, if I use MPICH for the MPI
> implementation.
>
> The hostfile specifies "slots=4 max-slots=4" for each host (trying to
> lie and say "slots=1" die not help), and I use "-np 4 --bynode" to get
> only one MPI process per host. I'm using ssh over Gbit ethernet
> between hosts.
>
> There is no useful error message that I can see. Watching top, I can
> see that processes are spawned on the four hosts, split into 4 OpenMP
> threads, and then crash immediately. The only error message is:
>
> mpirun noticed that job rank 0 with PID 30243 on node "coma006"
> exited on signal 11.
> Broken pipe

It looks like your application is dying from a segmentation fault.
The question is -- did Open MPI cause the segfault or is there
something in your application that Open MPI didn't like. It would be
useful to get a stack trace from the process that is causing the
segfault. Since you're only running 4 processes and using ssh to
start them, the easiest way is to start your process in gdb in an
xterm. You have to have ssh X forwarding enabled for this trick to
work, but then running something like:

     mpirun -np 4 --bynode -d xterm -e gdb <myapp>

should pop up 4 xterm windows, one for each process. Type "run" in
each gdb process in the xterms and it should be off and running.

If this would be a major pain, the other option is to try the nightly
build of Open MPI from the trunk, as it will try to print a stack
trace when errors like the one above occur. But I would start with
trying the gdb method. Of course, if you have TotalView or another
parallel debugger, that would be even easier.

Brian

-- 
   Brian Barrett
   Open MPI developer
   http://www.open-mpi.org/