I am replying to this via the daily-digest message I got. Sorry it wasn't sooner... I didn't realize I was getting replies until I got the digest. Does anyone know how to change it so I get the emails as you all send them?
>>Unless your computation is so "embarrassingly parallel" that each process can operate from start to end totally independent from the others, you should expect such an overlap, but you didn't tell what computation you want to do.
I am doing a Red-black Gauss-Seidel algorithm. The ghost points are what I was trying to figure for moving this into the 3rd dimension. Thanks for adding some concrete-ness to my idea of exactly how much overhead is involved. The test domains I am computing on are on the order of 100*50*50 or so. This is why I am trying to limit the overhead of the MPI communication. I am in the process of finding out exactly how big the domains may become, so that I can adjust the algorithm accordingly.
>> PS - The same book "MPI the complete reference" has a thorough description of MPI types in Chapter 3.
You may want to create and use a MPI_TYPE_VECTOR with the appropriate count, blocklength, and stride, to exchange all the "0..Z" overlap slices in a single swoop.
(If I understood right, this is your main concern.)
This is exactly what I was trying to do before, but I soon discovered you have to have your arrays tightly packed for this to work. Using a static array it is fine, but for dynamically sized arrays, there has to be a separate malloc() in C for the space, and then an array of pointers to that defined space to be able to use three-bracket notation. Tricky that it works like that (at least it was to me).
>>Also, I wonder why you want to decompose on both X and Y ("pencils"), and not only X ("books"), which may give you a smaller/simpler domain decomposition and communication footprint.
Whether you can or cannot do this way depends on your computation, which I don't know about.
If I understand what you mean by pencils versus books, I don't think that will work for these types of calculations will it? Maybe a better explanation of what you mean by a pencil versus a book. If I was going to solve a sub-matrix of the XY plane for all Z-values, what is that considered? My plan was to share the appropriate top, bottom, left, and right neighbors (w.r.t. to the XY plane) but NOT the above/below neighbors (since I am calculating from 0..Z completely, in order).