Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] maximum size for read buffer in MPI_File_read/write
From: Rob Latham (robl_at_[hidden])
Date: 2011-09-27 16:01:39


On Thu, Sep 22, 2011 at 11:37:10PM +0200, German Hoecht wrote:
> Hello,
>
> MPI_File_read/write functions uses an integer to specify the size of
> the buffer, for instance:
> int MPI_File_read(MPI_File fh, void *buf, int count, MPI_Datatype
> datatype, MPI_Status *status)
> with:
> count Number of elements in buffer (integer).
> datatype Data type of each buffer element (handle).
>
> However, using the maximum value of 32 bytes integers:
> count = 2^31-1 = 2147483647 (and datatype = MPI_BYTE)
> MPI_file_read only reads 2^31-2^12 = 2147479552 bytes.
> This means that 4095 bytes are ignored.
>
> I am not aware of this specific limit for integers in (Open) MPI
> function calls. Is this supposed to be correct?

Hi. I'm the ROMIO maintainer. OpenMPI more or less rolls up ROMIO
into OpenMPI, so any problems with the MPI_File_* routines is in my
lap, not OpenMPI.

I'll be honest with you: i've not given any thought to just how big
the biggest request could be. The independent routines, especially
with a simple type like MPI_BYTE, are going to almost immediately call
the underlying posix read() or write() call.

I can confirm the behavior you observe with your test program.
Thanks much for providing one. I'll dig around but I cannot think of
something in ROMIO that would ignore these 4095 bytes. I do think
it's legal by the letter of the standard to read or write less than
requested. "Upon completion, the amount of data accessed by the
calling process is returned in a status."

Bravo to you for actually checking return values and the status. I
don't think many non-library codes do that :>

I should at least be able to explain the behavior, so I'll dig a bit.

in general, if you plot "i/o performance vs blocksize", every file
system tops out around several tens of megabytes. So, we have given
the advice to just split up this nearly 2 gb request into several 1 gb
requests.

==rob

-- 
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA