From: German Hoecht <email@example.com>
To: Open MPI Users <firstname.lastname@example.org>; Rob Latham <email@example.com>
Sent: Wednesday, 28 September 2011, 10:09
Subject: Re: [OMPI users] maximum size for read buffer in MPI_File_read/write
thanks for your comments. I understand that it's most probably not worth
the effort to find the actual reason.
Because I have to deal with very large files I preferred using
"std::numeric_limits<int>::max()" rather than a hard-coded value
to split the read in case an IO request exceeds this amount. (This is
not the usual case but can happen.)
So your advice to use a max IO buffer of 1GB is quite precious.
To be honest, I did not do the check before we observed strange
numbers... Usually, MPI/ROMIO read/write functions are very stable, the
concerned code has read several Terabytes in the meanwhile.
On 09/27/2011 10:01 PM, Rob Latham wrote:
> On Thu, Sep
22, 2011 at 11:37:10PM +0200, German Hoecht wrote:
>> MPI_File_read/write functions uses an integer to specify the size of
>> the buffer, for instance:
>> int MPI_File_read(MPI_File fh, void *buf, int count, MPI_Datatype
>> datatype, MPI_Status *status)
>> count Number of elements in buffer (integer).
>> datatype Data type of each buffer element (handle).
>> However, using the maximum value of 32 bytes integers:
>> count = 2^31-1 = 2147483647 (and datatype = MPI_BYTE)
>> MPI_file_read only reads 2^31-2^12 = 2147479552 bytes.
>> This means that 4095 bytes are ignored.
>> I am not aware of this specific limit for integers in (Open) MPI
>> function calls. Is this supposed to be correct?
> Hi. I'm the ROMIO maintainer.
OpenMPI more or less rolls up ROMIO
> into OpenMPI, so any problems with the MPI_File_* routines is in my
> lap, not OpenMPI.
> I'll be honest with you: i've not given any thought to just how big
> the biggest request could be. The independent routines, especially
> with a simple type like MPI_BYTE, are going to almost immediately call
> the underlying posix read() or write() call.
> I can confirm the behavior you observe with your test program.
> Thanks much for providing one. I'll dig around but I cannot think of
> something in ROMIO that would ignore these 4095 bytes. I do think
> it's legal by the letter of the standard to read or write less than
> requested. "Upon completion, the amount of data accessed by the
> calling process is returned in a status."
> Bravo to you for actually checking return values and the
> don't think many non-library codes do that :>
> I should at least be able to explain the behavior, so I'll dig a bit.
> in general, if you plot "i/o performance vs blocksize", every file
> system tops out around several tens of megabytes. So, we have given
> the advice to just split up this nearly 2 gb request into several 1 gb
users mailing firstname.lastname@example.org://www.open-mpi.org/mailman/listinfo.cgi/users