There are multiple answers possible here. One is related to the over-
subscription of your cluster, but I expect that there are at least 4
cores per node if you want to use the slots=4 option. The real
question is what is the communication pattern in this benchmark ? and
how this match the distribution of the processes you use ?
As a matter of fact, if when you have XX processes per node, and all
of them will try to send a message to a remote process (here remote
means on another node), then they will have to share the physical
Myrinet link, which of course will lead to lower global performances
when XX increase (from 1, to 2 and then 4). And this is true without
regard on how you use the MX driver (via the Open MPI MTL or BTL).
Open MPI provide 2 options to allow you to distribute the processes
based on different criteria. Try to use -bynode and -byslot to see if
this affect the overall performances.
On Mar 28, 2007, at 9:56 AM, Warner Yuen wrote:
> Curious performance when using OpenMPI 1.2 to run Amber 9 on my
> Xserve Xeon 5100 cluster. Each cluster node is a dual socket, dual-
> core system. The cluster is also running with Myrinet 2000 with MX.
> I'm just running some tests with one of Amber's benchmarks.
> It seems that my hostfiles effect the performance of the
> application. I tried variations of the hostfile to see what would
> happen. I did a straight mpirun with no mca options set using:
> "mpirun -np 32"
> variation 1: hostname
> real 0m35.391s
> variation 2: hostname slots=4
> real 0m45.698s
> variation 3: hostname slots=2
> real 0m38.761s
> It seems that the best performance I achieve is when I use
> variation 1 with only the hostname and execute the command:
> "mpirun --hostfile hostfile -np 32 <my_application>" . Its
> shockingly about 13% better performance than if I use the hostfile
> with a syntax of "hostname slots=4".
> I also tried variations of in my mpirun command, here are the times:
> straight mpirun with not mca options
> real 0m45.698s
> "-mca mpi_yield_when_idle 0"
> real 0m44.912s
> "-mca mtl mx -mca pml cm"
> real 0m45.002s
> Warner Yuen
> Scientific Computing Consultant
> Apple Computer
> email: wyuen_at_[hidden]
> Tel: 408.718.2859
> Fax: 408.715.0133
> users mailing list