Thanks for the patience and the clarification - much appreciated. In fact, I
have someone that needs to learn more about the code base, so I think I will
assign this to him. At the least, he will have to learn a lot more about the
I have no problem with modifying the pernode behavior to deal with the case
of someone specifying -np as you describe. It would be relatively easy to
check. As I understand it, you want the behavior to be:
1. if no -np is specified, launch one proc/node across entire allocation
2. if -np n is specified AND n is less than the number of allocated nodes,
then launch one proc/node up to the specified number. Of course, this is
identical to just doing -np n -bynode, but that's immaterial. ;-)
3. if -np n is specified AND n is greater than the number of allocated
nodes, error message and exit
4. add a -npernode n option that launches n procs/node, subject to the same
Can you confirm?
Finally, I think you misunderstood my comment about the MPI folks. Our
non-MPI users couldn't care less about commonality of command line arguments
across MPI implementations. Hence, I leave issues in that area to the MPI
members of the team - they are the ones that decide how to deal with the
myriad of different option syntaxes in the MPI community.
Gives me too much of a headache! :-)
On 11/28/06 10:58 PM, "Maestas, Christopher Daniel" <cdmaest_at_[hidden]>
> Thanks for the feedback. Glad we are clearing these things up. :-)
> So here's what osc mpiexec is doing now:
> -pernode : allocate only one process per compute node
> -npernode <nprocs> : allocate no more than <nprocs> processes per
> compute node
>> Cdm> I think I originally requested the -pernode feature. :-) I've
>> Cdm> seen
>> one issue I know of ...
>> When used in the following manner:
>> "mpiexec -pernode -np N" and if N is > allocated nodes, it should
>> error out and not proceed. I need to update/learn to update the trac
>> ticket on this.
> This is an incorrect usage - the "pernode" option is only intended to be
> used without any specification of the number of processes. Pernode
> instructs the system to spawn one process/node across the entire
> allocation - we simply ignore any attempt to indicate the number of
> I suppose I could check and error out if you specify the number of procs
> AND --pernode. I would have to check the code, to be honest - I may
> already be doing so. Just don't remember :-)
> CDM> I think I remember looking through the code here, and thinking that
> app->num_procs needed to be compared to the
> opal_list_get_size(&master_node_list) given, but didn't dig into how
> that got set -np when specified. My intention is that we could do for
> an N node job scheduled allocation such as torque:
> "mpiexec -pernode myprogram"
> "mpiexec -np n -pernode myprogram" where n <= N
> That is how I believe osc mpiexec is behaving. I tested on 2 nodes and
> saw this:
> $ mpiexec -comm=none -pernode hostname
> $ mpiexec -comm=none -pernode -np 1 hostname
> $ mpiexec -comm=none -pernode -np 3 hostname
> mpiexec: Error: constrain_nodes: argument -n specifies 3 processors, but
> only 2 available after processing -pernode flag.
> This was my original intention of requesting -pernode in the first
> place. I apologize if I didn't provide this example as well. :-)
> Outside of a job scheduler I would think in that cause you would have
> the following to launch commands do the same thing:
> "mpiexec -np n -pernode --machinefile=m myprogram "
> "mpiexec -np n --bynode --machinefile=m"
> "OMPI_MCA_rmaps_base_schedule_policy=node; mpiexec -np n --machinefile=m
> Here the nodes N is basically "wc -l" of the file m and n <=N still
> holds true when using -pernode. It may prove difficult to check if
> using -pernode, and check in w/ all orted's to see if they've already
> launched the process on their nodes. I think that would have to be done
> if using -pernode though. If it's too difficult simply error'ng out
> when using -pernode and -np may be an easier choice for now. :-)
>> Cdm> it was more my hope that OSC mpiexec and Open MPI mpiexec options
>> would eventually merge into common options. A guy can dream can't he?
> Guess that's up to the MPI folks on the team - from an RTE perspective,
> I don't have an opinion either way.
> CDM> I would think that the mpiexec/orterun feature set could launch
> other types of executables that weren't mpi based, so I wouldn't think
> it would be just mpi spawning. :-)
> For example:
> "mpiexec -pernode hostname"
> users mailing list