Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] process placement with toruqe and OpenMP
From: Tim Prins (tprins_at_[hidden])
Date: 2008-02-07 12:04:55


Hi Brock,

As far as I know there is no way to do this with Open MPI and torque. I
believe people usually use hostfiles to do this sort of thing, but
hostfiles do not work with torque.

You may want to look into the launcher commands to see if torque will do
it for you. Slurm has an option '--cpus-per-task', but I just realized
we don't handle that properly...

Tim

Brock Palen wrote:
> Ok so I ask the mpirun masters how would you do the following:
>
> I submit a job with torque (we use --with-tm) like the following:
>
> nodes=4:ppn=2
>
> My desired outcome is to have 1 mpi process per 2 cpus and use
> threaded blas (or my own OpenMP take your pick)
>
> Our cluster has some 4 core machines thus the above job sometimes
> ends up looking like
>
> nodes=1:ppn=4+nodes=2:ppn=2
>
> The mpirun -bynode command will work in the case i get 4 nodes with
> only 2 cpus free. But if any machine other than the first machine is
> my node with 4 cores free given to me by moab, I would end up
> starting a extra process on the first node, where mpirun thinks
> another cpu is free, but that cpu is really to be used by OpenMP, and
> that the last process should be placed on the node that has 4 cpus free.
>
> I hope that wasn't to confusing, Its how to i launch hybrid jobs and
> make sure the process started by mpirun go where i want when my nodes
> have different core counts, and I am running via torque so using -H
> wont work, Also would prefer that all processes be started via TM.
>
> Is this posable ?
>
>
> Brock Palen
> Center for Advanced Computing
> brockp_at_[hidden]
> (734)936-1985
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users