Are there multiple interfaces on your nodes? I'm wondering if we are using a different network than the one where you opened these ports.

You'll get quite a bit of output, but you can turn on debug output in the oob itself with -mca oob_tcp_verbose xx. The higher the number, the more you get.


On Jul 10, 2010, at 11:14 AM, Robert Walters wrote:

Hello again,

I believe my administrator has opened the ports I requested. The problem I am having now is that OpenMPI is not listening to my defined port assignments in openmpi-mca-params.conf (looks like permission 644 on those files should it be 755?)

When I perform netstat -ltnup I see that orted is listening 14 processes in tcp but scaterred in the 26000ish port range when I specified 60001-60016 in the mca-params file. Is there a parameter I am missing? In any case I am still hanging as mentioned originally even with the port forwarding enabled and specifications in mca-param enabled.

Any other ideas on what might be causing the hang? Is there a more verbose mode I can employ to see more deeply into the issue? I have run --debug-daemons and --mca plm_base_verbose 99.

Thanks!
--- On Tue, 7/6/10, Robert Walters <raw19896@yahoo.com> wrote:

From: Robert Walters <raw19896@yahoo.com>
Subject: Re: [OMPI users] OpenMPI Hangs, No Error
To: "Open MPI Users" <users@open-mpi.org>
Date: Tuesday, July 6, 2010, 5:41 PM

Thanks for your expeditious responses, Ralph.

Just to confirm with you, I should change openmpi-mca-params.conf to include:

oob_tcp_port_min_v4 = (My minimum port in the range)
oob_tcp_port_range_v4 = (My port range)
btl_tcp_port_min_v4 = (My minimum port in the range)
btl_tcp_port_range_v4 = (My port range)

correct?

Also, for a cluster of around 32-64 processes (8 processors per node), how wide of a range will I require? I've noticed some entries in the mailing list suggesting you need a few to get started and then it opens as necessary. Will I be safe with 20 or should I go for 100?

Thanks again for all of your help!

--- On Tue, 7/6/10, Ralph Castain <rhc@open-mpi.org> wrote:

From: Ralph Castain <rhc@open-mpi.org>
Subject: Re: [OMPI users] OpenMPI Hangs, No Error
To: "Open MPI Users" <users@open-mpi.org>
Date: Tuesday, July 6, 2010, 5:31 PM

Problem isn't with ssh - the problem is that the daemons need to open a TCP connection back to the machine where mpirun is running. If the firewall blocks that connection, then we can't run.

If you can get a range of ports opened, then you can specify the ports OMPI should use for this purpose. If the sysadmin won't allow even that, then you are pretty well hosed.


On Jul 6, 2010, at 2:23 PM, Robert Walters wrote:

Yes, there is a system firewall. I don't think the sysadmin will allow it to go disabled. Each Linux machine has the built-in RHEL firewall. SSH is enabled through the firewall though.

--- On Tue, 7/6/10, Ralph Castain <rhc@open-mpi.org> wrote:

From: Ralph Castain <rhc@open-mpi.org>
Subject: Re: [OMPI users] OpenMPI Hangs, No Error
To: "Open MPI Users" <users@open-mpi.org>
Date: Tuesday, July 6, 2010, 4:19 PM

It looks like the remote daemon is starting - is there a firewall in the way?

On Jul 6, 2010, at 2:04 PM, Robert Walters wrote:

Hello all,

I am using OpenMPI 1.4.2 on RHEL. I have a cluster of AMD Opteron's and right now I am just working on getting OpenMPI itself up and running. I have a successful configure and make all install. LD_LIBRARY_PATH and PATH variables were correctly edited. mpirun -np 8 hello_c successfully works on all machines. I have setup my two test machines with DSA key pairs that successfully work with each other.

The problem comes when I initiate my hostfile to attempt to communicate across machines. The hostfile is setup correctly with <host_name> <slots> <max-slots>. When running with all verbose options enabled "mpirun --mca plm_base_verbose 99 --debug-daemons --mca btl_base_verbose 30 --mca oob_base_verbose 99 --mca pml_base_verbose 99 -hostfile hostfile -np 16 hello_c" I receive the following text output.

[machine1:03578] mca: base: components_open: Looking for plm components
[machine1:03578] mca: base: components_open: opening plm components
[machine1:03578] mca: base: components_open: found loaded component rsh
[machine1:03578] mca: base: components_open: component rsh has no register function
[machine1:03578] mca: base: components_open: component rsh open function successful
[machine1:03578] mca: base: components_open: found loaded component slurm
[machine1:03578] mca: base: components_open: component slurm has no register function
[machine1:03578] mca: base: components_open: component slurm open function successful
[machine1:03578] mca:base:select: Auto-selecting plm components
[machine1:03578] mca:base:select:(  plm) Querying component [rsh]
[machine1:03578] mca:base:select:(  plm) Query of component [rsh] set priority to 10
[machine1:03578] mca:base:select:(  plm) Querying component [slurm]
[machine1:03578] mca:base:select:(  plm) Skipping component [slurm]. Query failed to return a module
[machine1:03578] mca:base:select:(  plm) Selected component [rsh]
[machine1:03578] mca: base: close: component slurm closed
[machine1:03578] mca: base: close: unloading component slurm
[machine1:03578] mca: base: components_open: Looking for oob components
[machine1:03578] mca: base: components_open: opening oob components
[machine1:03578] mca: base: components_open: found loaded component tcp
[machine1:03578] mca: base: components_open: component tcp has no register function
[machine1:03578] mca: base: components_open: component tcp open function successful
Daemon was launched on machine2- beginning to initialize
[machine2:01962] mca: base: components_open: Looking for oob components
[machine2:01962] mca: base: components_open: opening oob components
[machine2:01962] mca: base: components_open: found loaded component tcp
[machine2:01962] mca: base: components_open: component tcp has no register function
[machine2:01962] mca: base: components_open: component tcp open function successful
Daemon [[1418,0],1] checking in as pid 1962 on host machine2
Daemon [[1418,0],1] not using static ports

At this point the system hangs indefinitely. While running top on the machine2 terminal, I see several things come up briefly. These items are: sshd (root), tcsh (myuser), orted (myuser), and mcstransd (root). I was wondering if sshd needs to be initiated by myuser? It is currently turned off in sshd_config through UsePAM yes. This was setup by the sysadmin but it can be worked around if this is necessary.

So in summary, mpirun works on each machine individually, but hangs when initiated through a hostfile or with the -host flag. ./configure with defaults and --prefix. LD_LIBRARY_PATH and PATH set up correctly. Any help is appreciated. Thanks!


_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


-----Inline Attachment Follows-----

_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


-----Inline Attachment Follows-----

_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users