Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] How are the Open MPI processes spawned?
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2011-11-28 19:32:27

On Nov 28, 2011, at 6:56 PM, Ralph Castain wrote:

> I'm afraid that example is incorrect - you were running under slurm on your cluster, not rsh.

Ummm... right. Duh.

> If you look at the actual code, you will see that we slurp up the envars into the environment of each app_context, and then send that to the backend.

Ah, right -- here's running under rsh (SVN trunk):

[16:26] svbu-mpi:~ % cat run
#!/bin/csh -f

echo on `hostname`, foo is: $OMPI_MCA_foo
exit 0
[16:26] svbu-mpi:~ % mpirun -np 2 --host svbu-mpi043,svbu-mpi044 run
OMPI_MCA_foo: Undefined variable.
OMPI_MCA_foo: Undefined variable.
While the primary job terminated normally, 2 processes returned
non-zero exit codes.. Further examination may be required.
[16:26] svbu-mpi:~ % setenv OMPI_MCA_foo bar
[16:27] svbu-mpi:~ % mpirun -np 2 --host svbu-mpi043,svbu-mpi044 run
on svbu-mpi044, foo is: bar
on svbu-mpi043, foo is: bar
[16:27] svbu-mpi:~ %

(the "MCA_" here is superfluous -- I looked at the code and the man page that Paul cited, and see that we grab all env vars OMPI_*, not OMPI_MCA_*).

> In environments like slurm, we can also apply those envars to the launch of the orteds as we pass the env to the API that launches the orteds. You cannot do that with rsh, as you know.

Right-o. Knew there was something I forgot...

> So on rsh, we do not put envar mca params onto the orted cmd line. This has been noted repeatedly on the user and devel lists, so it really has always been the case.

So they're sent as part of the launch command (i.e., out of band -- not on the rsh/ssh command line), right?

Meaning that per my output from above, what Paul was trying should have worked, no? I.e., setenv'ing OMPI_<whatever>, and those env vars should magically show up in the launched process.

>>>> [after performing some RTFM...]
>>>> at least the man page of mpiexec says, the OMPI_ environment variables are always provided and thus treated *differently* than other envvars:
>>>> $ man mpiexec
>>>> ....
>>>> Exported Environment Variables
>>>> All environment variables that are named in the form OMPI_* will automatically be exported to new processes on the local and remote nodes.
>>>> So, tells the man page lies, or this is an removed feature, or something else?

Jeff Squyres
For corporate legal information go to: