Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] OpenMPI Hangs, No Error
From: David Zhang (solarbikedz_at_[hidden])
Date: 2010-07-13 21:42:07


Like Ralph says, the slow down may not be coming from the kernel, but rather
on waiting for messages. What MPI send/recv commands are you using?

On Tue, Jul 13, 2010 at 11:53 AM, Ralph Castain <rhc_at_[hidden]> wrote:

> I'm afraid that having 2 cores on a single machine will always outperform
> having 1 core on each machine if any communication is involved.
>
> The most likely thing that is happening is that OMPI is polling waiting for
> messages to arrive. You might look closer at your code to try and optimize
> it better so that number-crunching can get more attention.
>
> Others on this list are far more knowledgeable than I am about doing such
> things, so I'll let them take it from here. Glad it is now running!
>
>
> On Jul 13, 2010, at 12:22 PM, Robert Walters wrote:
>
> OpenMPI,
>
> Following up. The sysadmin opened ports for machine to machine
> communication and OpenMPI is running successfully with no errors in
> connectivity_c, hello_c, or ring_c. Since, I have started to implement our
> MPP software (finite element analysis) that we have, and upon running a
> simple, 1 core on machine1, 1 core on machine2, job, I notice it is
> considerably slower than a 2 core job on a single machine.
>
> A quick look at top shows me kernel usage is almost twice what cpu usage
> is! On a 16 core job, (8 cores per node so 2 nodes total) test, OpenMPI was
> consuming ~65% of the cpu for kernel related items rather than
> number-crunching related items...Granted, we are running on GigE, but this
> is a finite element code we are running with no heavy data transfer within
> it. I'm looking into benchmarking tools, but my sysadmin is not very open to
> installing third party softwares. Do you have any suggestions for what I can
> use that would be "big name" or guaranteed safe tools I can use to figure
> out what's causing the hold up with all the kernel usage? I'm pretty sure
> its network traffic but I have no way of telling (as far as I know because
> I'm not a Linux whiz) with the standard tools in RHEL.
>
> Thanks for all the help! I'm glad to get it finally working and I think
> with a little tweaking it should be ready to go very soon.
>
> Regards,
> Robert Walters
> --- On *Sat, 7/10/10, Ralph Castain <rhc_at_[hidden]>* wrote:
>
>
> From: Ralph Castain <rhc_at_[hidden]>
> Subject: Re: [OMPI users] OpenMPI Hangs, No Error
> To: "Open MPI Users" <users_at_[hidden]>
> Date: Saturday, July 10, 2010, 4:37 PM
>
> The "static ports" flag means something different - it is used when the
> daemon is given a fixed port to use. In some installations, we lock every
> daemon to the same port number so that each daemon can compute exactly how
> to contact its peers (i.e., no contact info exchange required for wireup).
>
> You have a "fixed range", but not "static port", scenario. Hence the
> message.
>
> Let us know how it goes - I agree it sounds like something to discuss with
> the sysadmin.
>
>
> On Jul 10, 2010, at 1:47 PM, Robert Walters wrote:
>
> I ran oob_tcp_verbose 99 and I am getting something interesting I never got
> before.
>
> [machine 2:22347] bind() failed: no port available in the range
> [60001-60016]
> [machine 2:22347] mca_oob_tcp_init: unable to create IPv4 listen socket:
> Error
>
> I never got that error before we messed with the iptables but now I get
> that error... Very interesting, I will have to talk to my sysadmin again and
> make sure he opened the right ports on my two test machines. It looks as
> though there are no open ports. Another interesting thing is I see that the
> Daemon is still report:
>
> Daemon [[28845,0],1] checking in as pid 22347 on host machine 2
> Daemon [[28845,0],1] not using static ports
>
> Which, I may be misunderstanding, should have been taken care of when I
> specified what ports to use. I am telling it a static set of ports...
> Anyhow, I will get with my sysadmin again and see what he says. At least
> OpenMPI is correctly interpreting the range.
>
> Thanks for the help.
>
> --- On *Sat, 7/10/10, Ralph Castain <rhc_at_[hidden]>* wrote:
>
>
> From: Ralph Castain <rhc_at_[hidden]>
> Subject: Re: [OMPI users] OpenMPI Hangs, No Error
> To: "Open MPI Users" <users_at_[hidden]>
> Date: Saturday, July 10, 2010, 3:21 PM
>
> Are there multiple interfaces on your nodes? I'm wondering if we are using
> a different network than the one where you opened these ports.
>
> You'll get quite a bit of output, but you can turn on debug output in the
> oob itself with -mca oob_tcp_verbose xx. The higher the number, the more you
> get.
>
>
> On Jul 10, 2010, at 11:14 AM, Robert Walters wrote:
>
> Hello again,
>
> I believe my administrator has opened the ports I requested. The problem I
> am having now is that OpenMPI is not listening to my defined port
> assignments in openmpi-mca-params.conf (looks like permission 644 on those
> files should it be 755?)
>
> When I perform netstat -ltnup I see that orted is listening 14 processes in
> tcp but scaterred in the 26000ish port range when I specified 60001-60016 in
> the mca-params file. Is there a parameter I am missing? In any case I am
> still hanging as mentioned originally even with the port forwarding enabled
> and specifications in mca-param enabled.
>
> Any other ideas on what might be causing the hang? Is there a more verbose
> mode I can employ to see more deeply into the issue? I have run
> --debug-daemons and --mca plm_base_verbose 99.
>
> Thanks!
> --- On *Tue, 7/6/10, Robert Walters <raw19896_at_[hidden]>* wrote:
>
>
> From: Robert Walters <raw19896_at_[hidden]>
> Subject: Re: [OMPI users] OpenMPI Hangs, No Error
> To: "Open MPI Users" <users_at_[hidden]>
> Date: Tuesday, July 6, 2010, 5:41 PM
>
> Thanks for your expeditious responses, Ralph.
>
> Just to confirm with you, I should change openmpi-mca-params.conf to
> include:
>
> oob_tcp_port_min_v4 = (My minimum port in the range)
> oob_tcp_port_range_v4 = (My port range)
> btl_tcp_port_min_v4 = (My minimum port in the range)
> btl_tcp_port_range_v4 = (My port range)
>
> correct?
>
> Also, for a cluster of around 32-64 processes (8 processors per node), how
> wide of a range will I require? I've noticed some entries in the mailing
> list suggesting you need a few to get started and then it opens as
> necessary. Will I be safe with 20 or should I go for 100?
>
> Thanks again for all of your help!
>
> --- On *Tue, 7/6/10, Ralph Castain <rhc_at_[hidden]>* wrote:
>
>
> From: Ralph Castain <rhc_at_[hidden]>
> Subject: Re: [OMPI users] OpenMPI Hangs, No Error
> To: "Open MPI Users" <users_at_[hidden]>
> Date: Tuesday, July 6, 2010, 5:31 PM
>
> Problem isn't with ssh - the problem is that the daemons need to open a TCP
> connection back to the machine where mpirun is running. If the firewall
> blocks that connection, then we can't run.
>
> If you can get a range of ports opened, then you can specify the ports OMPI
> should use for this purpose. If the sysadmin won't allow even that, then you
> are pretty well hosed.
>
>
> On Jul 6, 2010, at 2:23 PM, Robert Walters wrote:
>
> Yes, there is a system firewall. I don't think the sysadmin will allow it
> to go disabled. Each Linux machine has the built-in RHEL firewall. SSH is
> enabled through the firewall though.
>
> --- On *Tue, 7/6/10, Ralph Castain <rhc_at_[hidden]>* wrote:
>
>
> From: Ralph Castain <rhc_at_[hidden]>
> Subject: Re: [OMPI users] OpenMPI Hangs, No Error
> To: "Open MPI Users" <users_at_[hidden]>
> Date: Tuesday, July 6, 2010, 4:19 PM
>
> It looks like the remote daemon is starting - is there a firewall in the
> way?
>
> On Jul 6, 2010, at 2:04 PM, Robert Walters wrote:
>
> Hello all,
>
> I am using OpenMPI 1.4.2 on RHEL. I have a cluster of AMD Opteron's and
> right now I am just working on getting OpenMPI itself up and running. I have
> a successful configure and make all install. LD_LIBRARY_PATH and PATH
> variables were correctly edited. mpirun -np 8 hello_c successfully works on
> all machines. I have setup my two test machines with DSA key pairs that
> successfully work with each other.
>
> The problem comes when I initiate my hostfile to attempt to communicate
> across machines. The hostfile is setup correctly with <host_name> <slots>
> <max-slots>. When running with all verbose options enabled "mpirun --mca
> plm_base_verbose 99 --debug-daemons --mca btl_base_verbose 30 --mca
> oob_base_verbose 99 --mca pml_base_verbose 99 -hostfile hostfile -np 16
> hello_c" I receive the following text output.
>
> [machine1:03578] mca: base: components_open: Looking for plm components
> [machine1:03578] mca: base: components_open: opening plm components
> [machine1:03578] mca: base: components_open: found loaded component rsh
> [machine1:03578] mca: base: components_open: component rsh has no register
> function
> [machine1:03578] mca: base: components_open: component rsh open function
> successful
> [machine1:03578] mca: base: components_open: found loaded component slurm
> [machine1:03578] mca: base: components_open: component slurm has no
> register function
> [machine1:03578] mca: base: components_open: component slurm open function
> successful
> [machine1:03578] mca:base:select: Auto-selecting plm components
> [machine1:03578] mca:base:select:( plm) Querying component [rsh]
> [machine1:03578] mca:base:select:( plm) Query of component [rsh] set
> priority to 10
> [machine1:03578] mca:base:select:( plm) Querying component [slurm]
> [machine1:03578] mca:base:select:( plm) Skipping component [slurm]. Query
> failed to return a module
> [machine1:03578] mca:base:select:( plm) Selected component [rsh]
> [machine1:03578] mca: base: close: component slurm closed
> [machine1:03578] mca: base: close: unloading component slurm
> [machine1:03578] mca: base: components_open: Looking for oob components
> [machine1:03578] mca: base: components_open: opening oob components
> [machine1:03578] mca: base: components_open: found loaded component tcp
> [machine1:03578] mca: base: components_open: component tcp has no register
> function
> [machine1:03578] mca: base: components_open: component tcp open function
> successful
> Daemon was launched on machine2- beginning to initialize
> [machine2:01962] mca: base: components_open: Looking for oob components
> [machine2:01962] mca: base: components_open: opening oob components
> [machine2:01962] mca: base: components_open: found loaded component tcp
> [machine2:01962] mca: base: components_open: component tcp has no register
> function
> [machine2:01962] mca: base: components_open: component tcp open function
> successful
> Daemon [[1418,0],1] checking in as pid 1962 on host machine2
> Daemon [[1418,0],1] not using static ports
>
> At this point the system hangs indefinitely. While running top on the
> machine2 terminal, I see several things come up briefly. These items are:
> sshd (root), tcsh (myuser), orted (myuser), and mcstransd (root). I was
> wondering if sshd needs to be initiated by myuser? It is currently turned
> off in sshd_config through UsePAM yes. This was setup by the sysadmin but it
> can be worked around if this is necessary.
>
> So in summary, mpirun works on each machine individually, but hangs when
> initiated through a hostfile or with the -host flag. ./configure with
> defaults and --prefix. LD_LIBRARY_PATH and PATH set up correctly. Any help
> is appreciated. Thanks!
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> -----Inline Attachment Follows-----
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> -----Inline Attachment Follows-----
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> -----Inline Attachment Follows-----
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> -----Inline Attachment Follows-----
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

-- 
David Zhang
University of California, San Diego