Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] 3D domain decomposition with MPI
From: Jed Brown (jed_at_[hidden])
Date: 2010-03-11 05:44:37


On Wed, 10 Mar 2010 22:25:43 -0500, Gus Correa <gus_at_[hidden]> wrote:

> Ocean dynamics equations, at least in the codes I've seen,
> normally use "pencil" decomposition, and are probably harder to
> handle using 3D "chunk" decomposition (due to the asymmetry imposed by
> gravity).

There is also a lot to be said for the strength of coupling. Ocean
codes do "tridiagonal solves" in columns, and these would no longer be
trivially cheap (in fact the structure of the code would need to change)
if the partition also broke up the vertical. Since the domain is so
anisotropic and the coupling (at least aside from the barotropic mode)
is so much stronger in the vertical than the horizontal, it is good to
decompose with columns always kept together. In a fully implicit code,
these column solves would quit being mandatory, but the availability of
a "line smoother" for multigrid still favors this type of decomposition.

Also note that in domain decomposition algorithms (like additive
Schwarz, and balancing Neumann-Neumann), the asymptotics scale with the
maximum number of subdomains required to cross the domain, and/or with
the number of elements along the longest edge of a subdomain. This
tends to favor partitioning in 3D, unless the physics/domain is
sufficiently anisotropic to overcome this preference.

Also depending on Derek's application, he may want to use a library like
PETSc to handle the decomposition and updates. Certainly this is true
if the application may ever need solvers; in my opinion, it is also true
unless this is a toy problem being used to learn MPI. If you really
want to write this stuff yourself, it's still worth looking at the
discussion in PETSc user's manual.

Jed