On Jan 30, 2006, at 3:39 PM, Drew McCormack wrote:
> Hello Brian,
> I sent the email below several days ago. I thought maybe you had
> missed it, so I'm sending it again. Maybe you deliberately ignored
> it or didn't have time to answer. That's fine, feel free to ignore
> it again ;-)
Sorry about the slow response, but last week was our quarterly Open
MPI meeting. Responses to mailing lists are (unfortunately) a bit
slower when we're all spending really long days trying to improve
Open MPI :).
> Thanks for the information. I have been playing with OpenMPI and
> Xgrid a little this week, and hadn't had much luck. This email
> helps a lot.
>> The XGrid starter currently looks for a couple of environment
>> variables to decide if it can be used. Currently, the XGrid process
>> starter only supports the basic password authentication to the
>> controller. As such, the two environment variables the XGrid starter
>> looks for are XGRID_CONTROLLER_HOSTNAME and
>> XGRID_CONTROLLER_PASSWORD. These are the same environment variables
>> that the 'xgrid' command-line submission process uses.
> Do you mean on the client/submission machine, or the agent machines
> where the applications are run?
> I guess you mean the client, right?
> So, I guess I have to make sure I set these environment variables,
> rather than just using the -p and -h xgrid command options.
Correct, these should be set on the client / submission machine. The
agent machines have their own authentication mechanism to connect to
> The reason I am a little confused is that I am pretty sure with our
> other MPI implementations, that mpirun gets called on the
> computational node after the queueing system has started the job
> running. What you seem to be indicating is that mpirun replaces the
> queueing system call in this case, and is issued from the
> submission node.
Yes, this is one place that our XGrid support is a little different
than other batch queuing systems. At the time that our XGrid support
was written, XGrid did not provide a mechanism for running a script
in the traditional sense (like PBS or SGE might do). We might
eventually support a different submission mechanism to better allow
integration of Open MPI into user applications, but this is still in
the development phase.
We would love to hear any comments on how people want to use Open MPI
and XGrid together. I can't promise we'll implement all the ideas,
but it will help give us direction on where to spend our development
>> The restriction that Open MPI be installed on all nodes is a slightly
>> more difficult problem. Open MPI usually builds as a shared library
>> with a bunch of dynamically loaded shared objects, complicating the
>> list of what must be migrated. Even if statically linked, there is
>> still a helper process we have to migrate out with your application
>> (to deal with standard I/O in the expected way, along with some other
>> features that are much easier to implement with a helper daemon).
> I am happy to install OpenMPI everywhere at this point, but in the
> long run, it would be great to be able to run OpenMPI/Xgrid apps
> without requiring preinstalled components, even if the daemon needs
> to be sent via the network.
Yes - this is probably a year or so out, but it is something I would
like to implement. Of course, it would mean giving up the ability to
build Open MPI as a set of shared libraries, so some flexibility
would be lost in the process.
>> To use the XGrid system, make sure that the XGrid controller is
>> properly configured to use password-based authentication. Then
>> issues the following commands (assuming tcsh)
>> % setenv XGRID_CONTROLLER_HOSTNAME mycomputer.apple.com
>> % setenv XGRID_CONTROLLER_PASSWORD pword
>> % mpirun -np X ./myapp
> I am assuming this is from the client/submission machine. So mpirun
> replaces the xgrid command. I guess I never need to use the xgrid
> command for OpenMPI/Xgrid jobs (?)
Correct - you use mpirun instead of the xgrid command to submit jobs.
> If this is the case, my next question is, how do I supply the usual
> xgrid options, such as working directory, standard input file, etc?
> Or is that simply not possible?
> Do I simply have to have some other way (eg ssh) to get files to/
> from agent machines, like I would for a batch system like PBS?
It looks like I never implemented those options (shame on me). I've
added that to my to-do list, although I can't give an accurate time-
table for implementation at this point. One thing to note is that
rather than using XGrid's standard input/output forwarding services,
we use Open MPI's services. So if you do:
mpirun -np 2 ./myapp < foo.txt
the result will be that rank 0 in the job will be able to read the
contents of foo.txt from standard input. There is a small bug in
this that will be fixed in 1.0.2 (that applies to all scenarios, not
just XGrid). Also, all standard output / standard error is output
through mpirun's standard output / standard error, so you can do
mpirun -np 2 ./myapp < foo.txt > bar.txt
to redirect foo.txt into rank 0's standard input and redirect all
standard output from all ranks into the file bar.txt.
> If I can get it all working, I will write up a few instructions on
> my web site, which may take the pressure of you to generate some docs.
Thanks. I don't think it gets me off the hook with my boss ;), but
the more resources, the better for the Mac community.
Open MPI developer