Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Ralph Castain (rhc_at_[hidden])
Date: 2006-03-31 14:18:26


Hi Rolf

See below!

Rolf Vandevaart wrote:
Hi Ralph:
Thanks for your information.   You said I could ask more so I am!  See 
below.

Ralph Castain wrote On 03/30/06 16:51,:

  
Hi Rolf

I apologize for the scarce documentation - we are working on it, but 
have a ways to go. I've tried to address your questions below. Please 
feel free to ask more!

Ralph

Rolf Vandevaart wrote:

    
Greetings:
I am new to the Open MPI world, and I have been trying to get a better
understanding of the ORTE environment.  At this point, I have a few
questions that I was hoping someone could answer.

1. I have heard mention of running the ORTE daemons in persistent mode,
however, I can find no details of how to do this.  Are there arguments
to either orted or mpirun to make this work right?   
      
Normally, we start a persistent daemon with:
orted --seed --persistent --scope=public

This will start the daemon and "daemonize" it so it keeps running 
until told to die. The arguments worth noting are:

(a) --persistent. Tells the daemon to "stay alive" until specifically 
told to "die"

(b) --scope=[public, private, exclusive]. This actually pertains to 
the universe, but you'll need to provide it anyway to ensure proper 
connectivity to anything you try to run. Right now, the daemons 
default to "exclusive", which means nothing can connect to them except 
the application that spawned them - no value to anyone if started with 
the above command! Private would exclude them to contact only from you 
- I haven't tested this enough to guarantee its functionality. I 
usually run them as "public" since security isn't a big concern right 
now - all this means is that anyone who can read the session directory 
tree (which is normally "locked" to only you anyway) would be able to 
connect to the daemon.

(c) --seed. Indicates that this daemon is the first one and therefore 
will host the data storage for the registry and other central services

(d) --universe=userid@hostname:universe_name. Allows you to name your 
universe to whatever you like. We use this to allow you to have 
multiple universes co-existing but separate - I've been explaining the 
reasons for that elsewhere, but will send them to this list if 
desired. You don't have to provide this, nor do you have to provide 
all the fields (e.g., you could just say "--universe=foo" to set the 
universe name).

You can provide the same options to mpirun, if you like - mpirun will 
simply start an orted and pass those parameters along, and the orted 
will merrily stay alive after the specified application completes.

    
While I understand all that has been written here in theory, I am still 
struggling
to get things to work.

The persistent daemon seems to be ignored when I do an mpirun.  I have 
watched the
system calls and looked at the process tree, and the persistent daemon 
does not seem
to be part of the fun.  So, I will be specific about what I am doing, 
and maybe you can point
out what I am doing wrong.

I have a 3 node cluster.  ct2, ct4, and ct5.  I am launching the job 
from ct2 and trying to
run on ct4 and ct5 which have persistent daemons on them.  I have 
selected the daemon
on ct4 to be the seed.

ct4> orted --seed --persistent --scope public -universe foo
ct5> orted --persistent --scope public -universe foo
ct2> mpirun --mca pls_rsh_agent rsh -np 4 -host ct4,ct5 -universe foo 
my_connectivity -v

While the program is running, I see this on ct4 and ct5.

ps -ef | grep orted
   rolfv  9456     1   0 11:24:26 ?           0:00 orted --bootproxy 1 
--name 0.0.2 --num_procs 3 --vpid_start 0 --nodename ct4
   rolfv  9386     1   0 11:21:30 ?           0:00 orted --seed 
--persistent --scope public --universe foo
 
Thanks for any additional details.


  
I see the problem - really just a current limit on our implementation. At the moment, our launchers don't know how to take advantage of existing daemons on the remote nodes. Your mpirun is correctly connecting to the persistent daemon on ct2, but will launch its own daemons on the other nodes.

This is a known issue that we need to address - just low on the priority list right now.

*snip*

  
3. I have a similar question about orteprobe.  Is this something
we should know about?
  
      
Yes and no - there's nothing secret about it. We use it internally to 
OpenRTE to "probe" a machine and see if we have a daemon/universe 
operating on it. Basically, we launch orteprobe on the remote machine 
- it checks to see if a session directory exists on it, attempts to 
connect to any universes it finds, and then reports back on its 
findings. Based on that report, we either launch an orted on the 
remote machine (to act as our surrogate so we can launch an 
application on that cell) or connect to an existing universe on the 
remote machine (and then tell it to launch the application for us).

    
4. Is there an easy way to view the data in the General Purpose
Registry?  This may be related to my first question, in that I
could imagine having persistent daemons and then I would like
to see what is stored in the registry.
  
      
Well, yes and no. Ideally, that would be a command from within the 
orteconsole function, but I don't think that has been implemented yet. 
I'd be happy to do so, if that is something you would like (shouldn't 
take long at all). There are a set of "dump" functions in the registry 
API for just that purpose. I usually access them via gdb - I attach 
the debugger to the orted process, then use the dump functions to 
output the values in the registry.
    

What exactly do you type in for the dump functions?  I saw these functions,
but could not get them to fire properly.

*snip*

  
Tell you what I'll do - I'll implement an interface in orteconsole that will let you execute the dump commands from there. The problem with doing it from gdb is that gdb doesn't like the volume of text that can result - so it keeps cutting off the dump output. Also, gdb - particularly if you are using shared libraries instead of a static build configuration - can bark when you try to access the registry functions directly, depending upon when exactly you do so.

The orteconsole approach will be *much* cleaner. Only negative is that you won't be able to look at the contents step-by-step as the application progresses.

If you really want to do it from gdb, then you have to type in the name of the replica's dump functions (e.g., orte_gpr_replica_dump_all) or the proxy's dump functions (e.g., orte_gpr_proxy_dump_all) directly - you can't go through the orte_gpr.xxx interface. Obviously, you use the replica commands on the persistent daemon (you'll probably want to tell it --no-daemonize so gdb can remain connected to it) and the proxy commands if you are working on one of the application processes.

Doing this will allow you to step through the process launch progression within mpirun (or the persistent daemon, if you are using it) and see what info is being stored where/when. Like I said, though, gdb will truncate the output, so you won't see everything :-(

Regards,
Rolf