Hi Roman
Just a guess.
Is this a domain decomposition code?
(I never heard about "cpmd 32 waters" before, sorry.)
Is it based on finite differences, finite volume, finite element?
If it is, once the size of the subdomains becomes too small compared to
the size of the halo around them, the overhead required to calculate
your solution for the halo swamps the whole calculation,
and scaling degrades.
This is not an MPI scaling problem, this is intrinsic to the domain
decomposition technique.
Typically this happens as the number of processors reach some high
number (which depends on the size of the problem).
So, what you are seeing may not be a problem with OpenMPI scaling,
but just that your problem is not large enough to require the use of,
say, 48 or 64 processors.
For instance, imagine a 1D problem with a grid with 1024 points,
that require a 2 grid point overlap (halo) on the left and right
of any subdomain to be calculated in parallel (i.e. decomposing the
domain in parts).
If you divide the domain across two processors only, each processor
has to work not on 1024/2=512 points, but on 512+2+2=516 points.
The calculation on the two processors gets an overhead of 2*(2+2)=8 grid
points,w.r.t. the same calculation done on a single processor.
This is an overhead of 8/1024=0.8% only, so using 2 processors
will speedup the calculation by a factor close to 2 (but slightly lower).
However, if you divide the same problem across 64 subdomains (i.e 64
processors), the size of each subdomain is 1024/64=16,
plus 2 halo grid point on each side, i.e. 20 grid points.
So the overhead is much higher now, 4/16=25%.
Dividing the problem across 64 processors will not speed it up by
a factor of 64, but by much less.
Every domain decomposition program that we have here shows this
effect. If we give them more processors they scale well, up
to a point (say 16 or 32 processors, for a reasonably sized problem).
However, beyond that point the scaling slowly flattens out.
When you go and look at the grid size and the
large number of processors,
you realize that most of the effort is being done to calculate halos,
i.e. on overhead.
On top of that, there is the overhead due to MPI communication, of
course, but it is likely that the halo overhead is the dominant factor.
I would guess other classes of problems and parallel methods of solution
also have the same problem that domain decomposition shows.
Is this perhaps what is going on with your test code?
Take a look at the code to see what it is doing,
and in particularly, what is the problem size.
See if it really makes sense to distribute it over 64 processors,
of if a smaller number would be the right choice.
Also, if the program allows you to change the problem size,
try the test again with a larger problem size
(say, two or four times bigger),
and then go up to a large number of processors also.
With a larger problem size the scaling may be better too
(but the runtimes will grow as well).
Finally, since you are using Infiniband, and I wonder if all the
nodes connect to each other with the same latency, or if some
pairs of nodes have higher latency to communicate.
On a single switch hopefully the latency is the same for all pairs of nodes.
However, if you connect two switches, for instance, nodes that
are on switch A will probably have a larger latency to talk
to nodes on switch B, I suppose.
I hope it helps.
Gus Correa

Gustavo Correa
LamontDoherty Earth Observatory  Columbia University
Palisades, NY, 109648000  USA

Roman Martonak wrote:
> Hello,
>
> I observe very poor scaling with openmpi on HP blade system consisting
> of 8 blades (each having 2 quadcore AMD Barcelona 2.2 GHz CPU) and
> interconnected with Infiniband fabric. When running the standard cpmd
> 32 waters test, I observe the following scaling (the numbers are
> elapsed time)
>
> openmpi1.2.6:
>
> using full blades (8 cores per blade)
> np8 7 MINUTES 26.40 SECONDS
> np16 4 MINUTES 19.91 SECONDS
> np32 2 MINUTES 55.51 SECONDS
> np48 2 MINUTES 38.18 SECONDS
> np64 3 MINUTES 19.78 SECONDS
>
> I tried also openmpi1.2.8 and openmpi1.3 and it is about the same,
> openmpi1.3 is somewhat better for 32 cores but in all cases there is
> practically no scaling beyond 4 blades (32 cores) and running on 64
> cores is a disaster. With Intel MPI, however, I get the following
> numbers
>
> Intel MPI3.2.1.009
>
> using full blades (8 cores per blade)
> np8 7 MINUTES 23.19 SECONDS
> np16 4 MINUTES 22.17 SECONDS
> np32 2 MINUTES 50.07 SECONDS
> np48 1 MINUTES 42.87 SECONDS
> np64 1 MINUTES 23.76 SECONDS
>
> so there is reasonably good scaling up to 64 cores. I am running with
> the option
> mca mpi_paffinity_alone 1, I have tried also mca btl_openib_use_srq
> 1 but it had only marginal effect. With mvapich I get similar scaling
> as with Intel MPI. The system is running the Rocksclusters
> distribution 5.1 with the mellanox ofed1.4 roll. I would be grateful
> if somebody could suggest me what could be the origin of the problem
> and how to tune openmpi to get better scaling.
>
> Many thanks in advance.
>
> Best regards
>
> Roman
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.openmpi.org/mailman/listinfo.cgi/users
