On Fri, Feb 03, 2012 at 10:46:21AM -0800, Tom Rosmond wrote:
> With all of this, here is my MPI related question. I recently added an
> option to use MPI-IO to do the heavy IO lifting in our applications. I
> would like to know what the relative importance of the dedicated MPI
> network vis-a-vis the GPFS network for typical MPIIO collective reads
> and writes. I assume there must be some hand-off of data between the
> networks during the process, but how is it done, and are there any rules
> to help understand it. Any insights would be welcome.
There's not really a handoff. MPI-IO on GPFS will call a posix read()
or write() system call after possibly doing some data massaging. That
system call sends data over the storage network.
If you've got a fast communication network but a slow storage network,
then some of the MPI-IO optimizations will need to be adjusted a bit.
Seems like you'd want to really beef up the "cb_buffer_size".
For GPFS, the big thing MPI-IO can do for you is align writes to
GPFS. see my next point.
> P.S. I am running with Open-mpi 1.4.2.
If you upgrade to something in the 1.5 series you will get some nice
ROMIO optimizations that will help you out with writes to GPFS if
you set the "striping_unit" hint to the GPFS block size.
Mathematics and Computer Science Division
Argonne National Lab, IL USA