Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] PATH settings
From: Martin Siegert (siegert_at_[hidden])
Date: 2011-09-24 21:53:05


Thanks, Jeff, for the details!

On Sat, Sep 24, 2011 at 07:26:49AM -0400, Jeff Squyres wrote:
> On Sep 22, 2011, at 11:06 PM, Martin Siegert wrote:
>
> > I am trying to figure out how openmpi (1.4.3) sets its PATH
> > for executables. From the man page:
> >
> > Locating Files
> > If no relative or absolute path is specified for a file, Open MPI will
> > first look for files by searching the directories specified by the
> > --path option. If there is no --path option set or if the file is not
> > found at the --path location, then Open MPI will search the user’s PATH
> > environment variable as defined on the source node(s).
>
> Oops -- it's not the source node, it's the running node. That being said, sometimes they're the same thing, and sometimes the PATH is copied (by the underlying run-time environment) to the target node.
>
> > This does not appear to be entirely correct - as far as I can tell
> > openmpi always prepends its own bin directory to the PATH before
> > searching for the executable. Can that be switched off?
>
> It should not be doing that unless you are specifying the full path name to mpirun, or using the --prefix option.

By now I recognize that my tests where flawed in in several aspects:
1) the path settings depend on whether you specify the full path to
mpiexec (as you mention), i.e., "/usr/local/openmpi/bin/mpiexec" does things
differently than "mpiexec" even though the executable is the same.
2) it makes a difference whether mpiexec runs from a torque batch
job or interactively (as you say below as well).

Nevertheless, I cannot avoid mpiexec prepending its own directory to the
PATH. This is what I tried:

dev:~> echo $PATH
/usr/local/bin:/usr/local/openmpi/bin:/usr/local/moab/bin:/usr/local/torque/bin:/bin:/usr/bin:/home/siegert/bin
# this is the default PATH on every node
dev:~> cat /home/siegert/scratch/test/path-0.0.1/bin/path.sh
#!/bin/bash
echo "`hostname`, $0:"
echo $PATH
dev:~> cat path.pbs
#!/bin/bash
#PBS -N path
#PBS -l walltime=1:00
#PBS -l nodes=2:ppn=1

export PATH=/home/siegert/scratch/test/path-0.0.1/bin:$PATH
echo $PATH
mpiexec path.sh
dev:~> qsub path.pbs
14.dev
dev:~> cat path.o14
/home/siegert/scratch/test/path-0.0.1/bin:/usr/local/bin:/usr/local/openmpi/bin:/usr/local/moab/bin:/usr/local/torque/bin:/bin:/usr/bin:/home/siegert/bin:/home/siegert/bin
b414, /home/siegert/scratch/test/path-0.0.1/bin/path.sh:
/usr/local/openmpi/bin:/usr/local/openmpi/bin:/home/siegert/scratch/test/path-0.0.1/bin:/usr/local/bin:/usr/local/openmpi/bin:/usr/local/moab/bin:/usr/local/torque/bin:/bin:/usr/bin:/home/siegert/bin:/home/siegert/bin
b413, /home/siegert/scratch/test/path-0.0.1/bin/path.sh:
/usr/local/openmpi/bin:/usr/local/openmpi/bin:/usr/local/openmpi/bin:/home/siegert/scratch/test/path-0.0.1/bin:/usr/local/bin:/usr/local/openmpi/bin:/usr/local/moab/bin:/usr/local/torque/bin:/bin:/usr/bin:/home/siegert/bin:/home/siegert/bin

Thus, on the local node (where mpiexec is run) /usr/local/openmpi/bin is
prepended twice, on the remote node /usr/local/openmpi/bin is prepended
three times.
But, this is the first point where I tricked myself: our "mpiexec" is a
wrapper script (in /usr/local/bin) that calls /usr/local/openmpi/bin/mpiexec:
dev:~> which mpiexec
/usr/local/bin/mpiexec
dev:~> which orterun
/usr/local/openmpi/bin/orterun

But, when I replace "mpiexec" in path.pbs with "orterun" the following
happens:

dev:~> cat path.pbs
#!/bin/bash
#PBS -N path
#PBS -l walltime=1:00
#PBS -l nodes=2:ppn=1

export PATH=/home/siegert/scratch/test/path-0.0.1/bin:$PATH
echo $PATH
orterun path.sh
dev:~> qsub path.pbs
15.dev
dev:~> cat path.o15
/home/siegert/scratch/test/path-0.0.1/bin:/usr/local/bin:/usr/local/openmpi/bin:/usr/local/moab/bin:/usr/local/torque/bin:/bin:/usr/bin:/home/siegert/bin:/home/siegert/bin
b414, /home/siegert/scratch/test/path-0.0.1/bin/path.sh:
/usr/local/openmpi-1.4.3/bin:/usr/local/openmpi-1.4.3/bin:/home/siegert/scratch/test/path-0.0.1/bin:/usr/local/bin:/usr/local/openmpi/bin:/usr/local/moab/bin:/usr/local/torque/bin:/bin:/usr/bin:/home/siegert/bin:/home/siegert/bin
b413, /home/siegert/scratch/test/path-0.0.1/bin/path.sh:
/usr/local/openmpi-1.4.3/bin:/usr/local/openmpi-1.4.3/bin:/usr/local/openmpi-1.4.3/bin:/home/siegert/scratch/test/path-0.0.1/bin:/usr/local/bin:/usr/local/openmpi/bin:/usr/local/moab/bin:/usr/local/torque/bin:/bin:/usr/bin:/home/siegert/bin:/home/siegert/bin

It appears that now "orterun" does something like "readlink -f $0":
/usr/local/openmpi is actually a softlink to /usr/local/openmpi-1.4.3.
Anyway, again the directory where the orterun executable is located
gets prepended twice on the local and three times on the remote node.
Only adding the --noprefix option to orterun avoids the prepending
of the directory (when calling "/usr/local/openmpi/bin/mpiexec --noprefix"
the --noprefix flag has no effect).

I guess, I could achieve what I want by using "orterun --noprefix" from the
wrapper script.

> > Furthermore, openmpi appears to use
> > a) the current value of PATH on the node where mpiexec is running;
> > b) whatever PATH is used by ssh on the remote nodes.
>
> mpirun uses the $PATH local to where it is. We don't ship the PATH to the remote node unless you tell mpirun to via the -x PATH option (as you noted below). We've found that default shipping the PATH to remote nodes can cause unexpected problems.
>
> That being said, some run-time systems (e.g., SLURM, Torque) automatically ship the front-end PATH to the back-end machine(s) for you. Open MPI just "inherits" this PATH on the remote node, so to speak. ssh doesn't do this by default.

Yup. That was the other way I tricked myself: trying to debug a bahaviour
when running under torque by running mpiexec interactively from the
head node. When "path.sh" is run interactively it fails because it is not
found on the remote node.

> Here's an example with 1.4.3 running SLURM on my test cluster at Cisco. This is in an SLURM allocation; I am running on the head node. Note that I'm a tcsh user, so I use "echo $path", not "echo $PATH":
>
> -----
> [4:23] svbu-mpi:~ % hostname
> svbu-mpi.cisco.com
> # Note my original path
> [4:23] svbu-mpi:~ % echo $path
> /users/jsquyres/local/rhel5/bin /home/jsquyres/bogus/bin /users/jsquyres/local/bin /usr/local/bin /users/jsquyres/local/rhel5/bin /home/jsquyres/bogus/bin /users/jsquyres/local/bin /usr/local/bin /usr/kerberos/bin /usr/local/bin /bin /usr/bin /usr/X11R6/bin /opt/slurm/2.1.0/bin /data/home/ted/bin /data/home/ted/bin
> # Since I'm in a SLURM allocation, mpirun sends jobs to a remote node
> [4:23] svbu-mpi:~ % mpirun -np 1 hostname
> svbu-mpi020
> # Here's my test script
> [4:23] svbu-mpi:~ % cat foo.csh
> #!/bin/tcsh -f
> echo $path
> # When I run this script through mpirun, the $path is the same
> # as was displayed above
> [4:23] svbu-mpi:~ % mpirun -np 1 foo.csh
> /users/jsquyres/local/rhel5/bin /home/jsquyres/bogus/bin /users/jsquyres/local/bin /usr/local/bin /users/jsquyres/local/rhel5/bin /home/jsquyres/bogus/bin /users/jsquyres/local/bin /usr/local/bin /usr/kerberos/bin /usr/local/bin /bin /usr/bin /usr/X11R6/bin /opt/slurm/2.1.0/bin /data/home/ted/bin /data/home/ted/bin
> # Now if I use the full path name to mpirun, I get an extra bonus
> # directory in the front of my $path -- the location of where
> # mpirun is located.
> [4:23] svbu-mpi:~ % /home/jsquyres/bogus/bin/mpirun -np 1 foo.csh
> /home/jsquyres/bogus/bin /home/jsquyres/bogus/bin /users/jsquyres/local/rhel5/bin /home/jsquyres/bogus/bin /users/jsquyres/local/bin /usr/local/bin /users/jsquyres/local/rhel5/bin /home/jsquyres/bogus/bin /users/jsquyres/local/bin /usr/local/bin /usr/kerberos/bin /usr/local/bin /bin /usr/bin /usr/X11R6/bin /opt/slurm/2.1.0/bin /data/home/ted/bin /data/home/ted/bin
> [4:23] svbu-mpi:~ %
> -----
>
> > Thus,
> >
> > export PATH=/path/to/special/bin:$PATH
> > mpiexec -n 2 -H n1,n2 special
> >
> > (n1 being the local node)
> > will usually fail even if the directory structure is identical on
> > the two nodes. For this to work
>
> The PATH you set will be available on n1, but it depends on the underlying run-time launcher if it is available on n2. ssh will not copy your PATH to n2 by default, but others will (e.g., SLURM).
>
> > mpiexec -n 2 -H n1,n2 -x PATH special
>
> That will work for ssh in this case, yes.
>
> > What I would like to see is a configure option that allows me to configure
> > openmpi such that the current PATH on the node where mpiexec is running
> > is used as the PATH on all nodes (by default). Or is there a reason why
> > that is a really bad idea?
>
> If your nodes are not exactly the same, this can lead to all kinds of badness. That's why we didn't do it by default.

I totally understand that you do not want to do this by default.
However, it would be nice to have a configure option like
--disable-prepend-ompi-path
that would at least prevent the prepending of the openmpi bin directory.
For those of us who do have identical nodes it would be even nicer to
have a configure option
--enable-path-propagation
that would always do -x PATH and not prepend the openmpi bin directory.

Cheers,
Martin

-- 
Martin Siegert
Simon Fraser University
Burnaby, British Columbia