Using direct can cause scaling issues as every process will open a socket to every other process in the job. You would at least have to ensure you have enough file descriptors available on every node.

The most likely cause is either (a) a different OMPI version getting picked up on one of the nodes, or (b) something blocking communication between at least one of your other nodes. I would suspect the latter - perhaps a firewall or something?

I''m disturbed by your not seeing any error output - that seems strange. Try adding --debug-daemons to the cmd line. That should definitely generate output from every daemon (at the least, they report they are alive).

Ralph

On Jul 29, 2009, at 2:06 PM, David Doria wrote:

On Wed, Jul 29, 2009 at 3:42 PM, Ralph Castain <rhc@open-mpi.org> wrote:
It sounds like perhaps IOF messages aren't getting relayed along the daemons. Note that the daemon on each node does have to be able to send TCP messages to all other nodes, not just mpirun.

Couple of things you can do to check:

1. -mca routed direct - this will send all messages direct instead of across the daemons

2. --leave-session-attached - will allow you to see any errors reported by the daemons, including those from attempting to relay messages

Ralph


Ralph, thanks for the quick response.

With
-mca routed direct
it works correctly.

With this:
mpirun -H 10.1.2.126,10.1.2.122,10.1.2.123,10.1.2.125 --leave-session-attached -np 4 /home/doriad/MPITest/hello-mpi

I still get no output nor errors from the daemons.

Is there a downside to using 'mca routed direct'? Or should I fix whatever is causing this daemon issue? You have any other tests for me to try to see what's wrong?

Thanks,

David
_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users