Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] very bad parallel scaling of vasp using openmpi
From: Joe Landman (landman_at_[hidden])
Date: 2009-08-18 09:39:52


Craig Plaisance wrote:
> Hi - I have compiled vasp 4.6.34 using the Intel fortran compiler 11.1
> with openmpi 1.3.3 on a cluster of 104 nodes running Rocks 5.2 with two
> quad core opterons connected by a Gbit ethernet. Running in parallel on

Latency of gigabit is likely your issue. Lower quality gigabit switches
often have very high port to port latencies.

NICs on motherboards also tend to be less ... well ... good at low
latency operations.

Size of your problem set is important ... small problems don't scale well.

You might try MX, and building OpenMPI atop MX, and see if this helps,
assuming problem size isn't the issue.

I'd also recommend (though this is hard) upgrading your node kernels.
Rocks uses a Redhat kernel, built on a fairly (ancient) 2.6.18 base,
with lots of backports. We normally suggest alternatives. Modern
kernels tend to be faster. Unfortunately, installing new kernels on
Rocks compute nodes can be a challenge.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman_at_[hidden]
web  : http://scalableinformatics.com
        http://scalableinformatics.com/jackrabbit
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615