We use Torque and Moab as our resource manager,
I have a user who has to mix programming models, threads and MPI, but not on all ranks,
Example:
Rank 0 needs 4 cores
Rank 1 needs 4 cores
Ranks 2-5 need 1
There is no way to give this to Torque/Moab in a way that when using TM openMPI correctly starts the layout I need.
I found I can use a rankfile to do this,
cat rankfile
rank 0=nyx0456 slot=0
rank 1=nyx0457 slot=0
rank 2=nyx0458 slot=0
rank 3=nyx0458 slot=1
rank 4=nyx0458 slot=2
rank 5=nyx0458 slot=3
qsub -I -l nodes=4:ppn=4
[brockp_at_nyx0456 tmp]$ mpirun -np 6 -rf rankfile hostname | sort | uniq -c
1 nyx0456.engin.umich.edu
1 nyx0457.engin.umich.edu
4 nyx0458.engin.umich.edu
(ignore the cpu count mismatch, I am not using all 16 cores in this example)
I could make a script to build this rankfile, I want to know if there is a better way to do this,
Also we don't want to use affinity, as we don't know what 'cpus' the bath (pbs/moab) system gave us, is the above rankfile going to do the right thing? That is the processes will float? Is there a way to tell if affinity is set? Or to say "rank 0 goes to this host use any available cores"
Thanks!
Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp_at_[hidden]
(734)936-1985
|
![[Flag]](../../../../../community/mirrors/flag-us.gif){kind=small}