Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Trouble with MPI-IO
From: Rob Latham (robl_at_[hidden])
Date: 2011-05-24 11:09:59


On Fri, May 20, 2011 at 08:14:07AM -0400, Jeff Squyres wrote:
> On May 20, 2011, at 6:23 AM, Jeff Squyres wrote:
>
> > Shouldn't ijlena and ijdisp be 1D arrays, not 2D arrays?
>
> Ok, if I convert ijlena and ijdisp to 1D arrays, I don't get the compile error (even though they're allocatable -- so allocate was a red herring, sorry). That's all that "use mpi" is complaining about -- that the function signatures didn't match.
>
> use mpi is your friend -- even if you don't use F90 constructs much. Compile-time checking is Very Good Thing (you were effectively "getting lucky" by passing in the 2D arrays, I think).
>
> Attached is my final version. And with this version, I see the hang when running it with the "T" parameter.
>
> That being said, I'm not an expert on the MPI IO stuff -- your code *looks* right to me, but I could be missing something subtle in the interpretation of MPI_FILE_SET_VIEW. I tried running your code with MPICH 1.3.2p1 and it also hung.
>
> Rob (ROMIO guy) -- can you comment this code? Is it correct?

There's a kind of obscure but important rule in MPI-IO: the file view
must describe monotonically non-decreasing offsets.

the T type creates a file type with the following flattened
representation (you can kind of think of the flattened representation
as a type map, except everything is in terms of bytes):

(0, 32), (96, 32), (32, 64)

So, 32 bytes at offset 0, 32 bytes at offset 96 and 64 bytes at offset
32.

That sort of looks like this:
|xxxx~~~~~~~~~~~~zzzz~~~~yyyy|

But you need the zzzz and yyyy pieces to be swapped in file view.

It's an annoying part of the standard but as you can see if you
violate that ROMIO will go off and spin in an infinite loop looking
for the next piece of I/O (which in this case was "behind" the current
piece).

You can work around this by adjusting your memory datatype: data must
be read off of the disk in this monotonically non-decreasing order but
it can be jammed into memory any which way you want.

ROMIO should be better about reporting file views that violate this
part of the standard. We report it in a few places but clearly not
enough.

==rob

-- 
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA