Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] using rankfiles to control mixed mode programming
From: Brock Palen (brockp_at_[hidden])
Date: 2010-06-16 16:21:20


We use Torque and Moab as our resource manager,
I have a user who has to mix programming models, threads and MPI, but not on all ranks,

Example:
Rank 0 needs 4 cores
Rank 1 needs 4 cores
Ranks 2-5 need 1

There is no way to give this to Torque/Moab in a way that when using TM openMPI correctly starts the layout I need.
I found I can use a rankfile to do this,

cat rankfile
rank 0=nyx0456 slot=0
rank 1=nyx0457 slot=0
rank 2=nyx0458 slot=0
rank 3=nyx0458 slot=1
rank 4=nyx0458 slot=2
rank 5=nyx0458 slot=3

qsub -I -l nodes=4:ppn=4
[brockp_at_nyx0456 tmp]$ mpirun -np 6 -rf rankfile hostname | sort | uniq -c
      1 nyx0456.engin.umich.edu
      1 nyx0457.engin.umich.edu
      4 nyx0458.engin.umich.edu

(ignore the cpu count mismatch, I am not using all 16 cores in this example)

I could make a script to build this rankfile, I want to know if there is a better way to do this,
Also we don't want to use affinity, as we don't know what 'cpus' the bath (pbs/moab) system gave us, is the above rankfile going to do the right thing? That is the processes will float? Is there a way to tell if affinity is set? Or to say "rank 0 goes to this host use any available cores"

Thanks!

Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp_at_[hidden]
(734)936-1985