Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] maximum size for read buffer in MPI_File_read/write
From: German Hoecht (german.hoecht_at_[hidden])
Date: 2011-09-28 04:09:58


Hi Rob,

thanks for your comments. I understand that it's most probably not worth
the effort to find the actual reason.

Because I have to deal with very large files I preferred using
"std::numeric_limits<int>::max()" rather than a hard-coded value
to split the read in case an IO request exceeds this amount. (This is
not the usual case but can happen.)

So your advice to use a max IO buffer of 1GB is quite precious.

To be honest, I did not do the check before we observed strange
numbers... Usually, MPI/ROMIO read/write functions are very stable, the
concerned code has read several Terabytes in the meanwhile.

Best regards,
German

On 09/27/2011 10:01 PM, Rob Latham wrote:
> On Thu, Sep 22, 2011 at 11:37:10PM +0200, German Hoecht wrote:
>> Hello,
>>
>> MPI_File_read/write functions uses an integer to specify the size of
>> the buffer, for instance:
>> int MPI_File_read(MPI_File fh, void *buf, int count, MPI_Datatype
>> datatype, MPI_Status *status)
>> with:
>> count Number of elements in buffer (integer).
>> datatype Data type of each buffer element (handle).
>>
>> However, using the maximum value of 32 bytes integers:
>> count = 2^31-1 = 2147483647 (and datatype = MPI_BYTE)
>> MPI_file_read only reads 2^31-2^12 = 2147479552 bytes.
>> This means that 4095 bytes are ignored.
>>
>> I am not aware of this specific limit for integers in (Open) MPI
>> function calls. Is this supposed to be correct?
>
> Hi. I'm the ROMIO maintainer. OpenMPI more or less rolls up ROMIO
> into OpenMPI, so any problems with the MPI_File_* routines is in my
> lap, not OpenMPI.
>
> I'll be honest with you: i've not given any thought to just how big
> the biggest request could be. The independent routines, especially
> with a simple type like MPI_BYTE, are going to almost immediately call
> the underlying posix read() or write() call.
>
> I can confirm the behavior you observe with your test program.
> Thanks much for providing one. I'll dig around but I cannot think of
> something in ROMIO that would ignore these 4095 bytes. I do think
> it's legal by the letter of the standard to read or write less than
> requested. "Upon completion, the amount of data accessed by the
> calling process is returned in a status."
>
> Bravo to you for actually checking return values and the status. I
> don't think many non-library codes do that :>
>
> I should at least be able to explain the behavior, so I'll dig a bit.
>
> in general, if you plot "i/o performance vs blocksize", every file
> system tops out around several tens of megabytes. So, we have given
> the advice to just split up this nearly 2 gb request into several 1 gb
> requests.
>
> ==rob
>