Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Maestas, Christopher Daniel (cdmaest_at_[hidden])
Date: 2006-11-29 00:58:51


Thanks for the feedback. Glad we are clearing these things up. :-)

So here's what osc mpiexec is doing now:

  -pernode : allocate only one process per compute node
  -npernode <nprocs> : allocate no more than <nprocs> processes per
compute node
> Cdm> I think I originally requested the -pernode feature. :-)  I've 
> Cdm> seen
> one issue I know of ...
> When used in the following manner:
> ---
> "mpiexec -pernode -np N" and if N is > allocated nodes, it should 
> error out and not proceed.  I need to update/learn to update the trac 
> ticket on this.
> ---
This is an incorrect usage - the "pernode" option is only intended to be
used without any specification of the number of processes. Pernode
instructs the system to spawn one process/node across the entire
allocation - we simply ignore any attempt to indicate the number of
I suppose I could check and error out if you specify the number of procs
AND --pernode. I would have to check the code, to be honest - I may
already be doing so. Just don't remember :-)
CDM> I think I remember looking through the code here, and thinking that
app->num_procs needed to be compared to the
opal_list_get_size(&master_node_list) given, but didn't dig into how
that got set -np when specified.  My intention is that we could do for
an N node job scheduled allocation such as torque:
"mpiexec -pernode myprogram"
"mpiexec -np n -pernode myprogram" where n <= N
That is how I believe osc mpiexec is behaving.  I tested on 2 nodes and
saw this:
$ mpiexec -comm=none -pernode hostname
$ mpiexec -comm=none -pernode -np 1 hostname
$ mpiexec -comm=none -pernode -np 3 hostname
mpiexec: Error: constrain_nodes: argument -n specifies 3 processors, but
  only 2 available after processing -pernode flag.
This was my original intention of requesting -pernode in the first
place.  I apologize if I didn't provide this example as well.  :-)
Outside of a job scheduler I would think in that cause you would have
the following to launch commands do the same thing:
"mpiexec -np n -pernode --machinefile=m myprogram "
"mpiexec -np n --bynode --machinefile=m"
"OMPI_MCA_rmaps_base_schedule_policy=node; mpiexec -np n --machinefile=m
Here the nodes N is basically "wc -l" of the file m and n <=N still
holds true when using -pernode.  It may prove difficult to check if
using -pernode, and check in w/ all orted's to see if they've already
launched the process on their nodes.  I think that would have to be done
if using -pernode though.  If it's too difficult simply error'ng out
when using -pernode and -np may be an easier choice for now. :-)  
> Cdm> it was more my hope that OSC mpiexec and Open MPI mpiexec options
> would eventually merge into common options.  A guy can dream can't he?
> ;-)
Guess that's up to the MPI folks on the team - from an RTE perspective,
I don't have an opinion either way.
CDM> I would think that the mpiexec/orterun feature set could launch
other types of executables that weren't mpi based, so I wouldn't think
it would be just mpi spawning. :-)
For example:
	"mpiexec -pernode hostname"