Using direct can cause scaling issues as every process will open a socket to every other process in the job. You would at least have to ensure you have enough file descriptors available on every node.
The most likely cause is either (a) a different OMPI version getting picked up on one of the nodes, or (b) something blocking communication between at least one of your other nodes. I would suspect the latter - perhaps a firewall or something?
I''m disturbed by your not seeing any error output - that seems strange. Try adding --debug-daemons to the cmd line. That should definitely generate output from every daemon (at the least, they report they are alive).
On Jul 29, 2009, at 2:06 PM, David Doria wrote:
On Wed, Jul 29, 2009 at 3:42 PM, Ralph Castain <firstname.lastname@example.org>
It sounds like perhaps IOF messages aren't getting relayed along the daemons. Note that the daemon on each node does have to be able to send TCP messages to all other nodes, not just mpirun.
Couple of things you can do to check:
1. -mca routed direct - this will send all messages direct instead of across the daemons
2. --leave-session-attached - will allow you to see any errors reported by the daemons, including those from attempting to relay messages
Ralph, thanks for the quick response.
-mca routed direct
it works correctly.
mpirun -H 10.1.2.126,10.1.2.122,10.1.2.123,10.1.2.125 --leave-session-attached -np 4 /home/doriad/MPITest/hello-mpi
I still get no output nor errors from the daemons.
Is there a downside to using 'mca routed direct'? Or should I fix whatever is causing this daemon issue? You have any other tests for me to try to see what's wrong?
users mailing list