Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] 3D domain decomposition with MPI
From: Eugene Loh (eugene.loh_at_[hidden])
Date: 2010-03-10 20:18:53

Gus Correa wrote:

> The redundant calculations of overlap points on neighbor subdomains
> in general cannot be avoided.
> Exchanging the overlap data across neighbor subdomain processes
> cannot be avoided either.
> However, **full overlap slices** are exchanged after each computational
> step (in our case here a time step).
> It is not a point-by-point exchange as you suggested.
> Overlap exchange does limit the usefulness/efficiency
> of using too many subdomains (e.g. if your overlap-to-useful-data
> ratio gets close to 100%).
> However, is not as detrimental as you imagined based on your
> point-by-point exchange conjecture.
> If your domain is 100x100x100 and you split in subdomain slices
> across 5 processes, with a 1-point overlap (on each side)
> you will have a 2x5/100 = 10% waste due to overlap calculations
> (plus the MPI communication cost/time),
> but your problem is still being solved in (almost) 1/5 of the time
> it would take in serial mode.

I don't understand what "waste" or "redundant calculations" you're
referring to here. Let's say each cell is updated based on itself and
neighboring values. Each cell has a unique "owner" who computes the
updated value. That value is shared with neighbors if the cell is near
the subdomain surface. So, there is a communication cost, but typically
each new value is computed by only one process.

E.g., say we have (in this extreme example) four values in a ring to be
distributed among two processes. So, P0 owns values 1 and 2 and P1 owns
values 3 and 4. In each iteration, boundary values are communicated.
In this particular, extreme example, that means that each process will
have all values. But then each process only updates/computes the values
it owns. There is, to be sure, a communication and synchronization
cost; but each value is computed by only one process.