Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2007-11-06 22:10:52


On Nov 2, 2007, at 11:02 AM, himanshu khandelia wrote:

> This question is about the use of a simulation package called GROMACS.
>
> PS: On our cluster (quad-core nodes), GROMACS does not scale well
> beyond 4 cpus. So, I wish to run two different simulations, while
> requesting 2 nodes (1 simulation on each node) to best exploit the
> policies of our MAUI scheduler.
>
> So, I am requesting 2 4-cpu nodes on a cluster using PBS. I want to
> run a separate simulation on each 4-cpu node. However, on 2 nodes, the
> speed for each simulation decreases (50 to 100%) if compared to a
> simulation which runs in a job which requests only one node. I am
> guessing this is because openmpi fails to assign all cpus of the same
> node to one simulation ? Instead, cpus from different nodes are being
> used to run simulation. This is what I have in the PBS script
>
> 1.
> ########
> mpirun -np 4 my-gromacs-executable-for-simulation-1 -np 4 &
> mpirun -np 4 my-gromacs-executable-for-simulation-1-np 4 &

In this case, Open MPI does not realize that you have executed 2
mpiruns and therefore assigns both the first and second job to the
same 4 processors. Hence, they run at half speed (or slower) because
they're competing for the same CPUs.

> # (THE GROMACS EXECUTABLE DOES REQUIRE A REPEAT REQUEST FOR THE NUMBER
> OF PROCESSORS)
>
> wait
> ########
>
> OPENMPI does have a mechanism whereby one can assign specific
> processes to specific nodes
>
> http://www.open-mpi.org/faq/?category=running#mpirun-scheduling
>
> So, I also tried all of the following in the PBS script where the
> --bynode or the --byslot option is used
>
> 2.
> ########
> mpirun -np 4 --bynode my-gromacs-executable-for-simulation-1 -np 4 &
> mpirun -np 4 --bynode my-gromacs-executable-for-simulation-2 -np 4 &
> wait
> ########
>
> 3.
> ########
> mpirun -np 4 --byslot my-gromacs-executable-for-simulation-1 -np 4 &
> mpirun -np 4 --byslot my-gromacs-executable-for-simulation-2 -np 4 &
> wait
> ########
>
> But these methods also result in similar performance losses.

The same thing will happen here -- OMPI is unaware of the 2 mpiruns,
and therefore schedules on the same first 4 nodes or slots.

It sounds like you really want to run two torque jobs, not one. Is
there a reason you're not doing that?

Failing that, you could do an end-run around the Open MPI torque
support and use the rsh launcher to precisely control where you launch
jobs, but that's a bunch of trouble and not really how we intended the
system to be used.

>
> So how does one assign the cpus properly using mpirun if running
> different simulations in the same PBS job ??
>
> Thank you for the help,
>
> -Himanshu
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

-- 
Jeff Squyres
Cisco Systems