Using direct can cause scaling issues as every process will open a
socket to every other process in the job. You would at least have to
ensure you have enough file descriptors available on every node.
The most likely cause is either (a) a different OMPI version getting
picked up on one of the nodes, or (b) something blocking communication
between at least one of your other nodes. I would suspect the latter -
perhaps a firewall or something?
I''m disturbed by your not seeing any error output - that seems
strange. Try adding --debug-daemons to the cmd line. That should
definitely generate output from every daemon (at the least, they
report they are alive).
On Jul 29, 2009, at 2:06 PM, David Doria wrote:
> On Wed, Jul 29, 2009 at 3:42 PM, Ralph Castain <rhc_at_[hidden]>
> It sounds like perhaps IOF messages aren't getting relayed along the
> daemons. Note that the daemon on each node does have to be able to
> send TCP messages to all other nodes, not just mpirun.
> Couple of things you can do to check:
> 1. -mca routed direct - this will send all messages direct instead
> of across the daemons
> 2. --leave-session-attached - will allow you to see any errors
> reported by the daemons, including those from attempting to relay
> Ralph, thanks for the quick response.
> -mca routed direct
> it works correctly.
> With this:
> mpirun -H 10.1.2.126,10.1.2.122,10.1.2.123,10.1.2.125 --leave-
> session-attached -np 4 /home/doriad/MPITest/hello-mpi
> I still get no output nor errors from the daemons.
> Is there a downside to using 'mca routed direct'? Or should I fix
> whatever is causing this daemon issue? You have any other tests for
> me to try to see what's wrong?
> users mailing list