Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Parallel file write in fortran (+mpi)
From: David Singleton (David.Singleton_at_[hidden])
Date: 2010-02-02 20:19:09


Its definitely not a bug in Lustre - its an essential part of any
"coherent" cluster filesystem (CXFS, QFS, GFS, GPFS, ....). The
whole point is that some people actually want to have meaningful
(non-garbage) files accessed using properly managed parallel IO
techniques. The locking is necessary in this case. You would see
the same issues on any of those filesystems.

David

On 02/03/2010 11:37 AM, Laurence Marks wrote:
> Agreed that it is not good (and I am recoding some programs to avoid
> this), but (and here life gets interesting) is this a "bug" in Lustre?
>
> On Tue, Feb 2, 2010 at 5:59 PM, David Singleton
> <David.Singleton_at_[hidden]> wrote:
>>
>> But its a very bad idea on a "coherent", "POSIX" filesystem like Lustre.
>> Locks have to bounce around between the nodes for every write. This can
>> be VERY slow (even for trivial amounts of "logging" IO) and thrash the
>> filesystem for other users. So, yes, at our site, we include this sort
>> of "parallel IO" on our list of disallowed behaviour. Not a good practice
>> to adopt in general.
>>
>> David
>>
>> On 02/03/2010 10:40 AM, Laurence Marks wrote:
>>>
>>> I know it's wrong, but I don't think it is forbidden.... (which I
>>> guess is what you are saying).
>>>
>>> On Tue, Feb 2, 2010 at 5:31 PM, Jeff Squyres<jsquyres_at_[hidden]> wrote:
>>>>
>>>> +1 on Nick's responses.
>>>>
>>>> AFAIK, if you don't mind getting garbage in the output file, it should be
>>>> fine to do. Specifically: it should not cause OS issues (crash, reboot,
>>>> corrupted filesystem, etc.) to do this -- but the file contents will likely
>>>> be garbage.
>>>>
>>>> That being said, this situation likely falls into the "Doc, it hurts when
>>>> I do this..." category. Meaning: you know it's wrong, so you probably
>>>> shouldn't be doing it anyway. :-)
>>>>
>>>>
>>>> On Feb 2, 2010, at 4:50 PM, Nicolas Bock wrote:
>>>>
>>>>> Hi Laurence,
>>>>>
>>>>> I don't know whether it's as bad as a deadly sin, but for us parallel
>>>>> writes are a huge problem and we get complete garbage in the file. Take a
>>>>> look at:
>>>>>
>>>>> Implementing MPI-IO Atomic Mode and Shared File Pointers Using MPI
>>>>> One-Sided Communication, Robert Latham,Robert Ross, Rajeev Thakur,
>>>>> International Journal of High Performance Computing Applications, 21, 132
>>>>> (2007).
>>>>>
>>>>> They describe an implemenation of a "mutex" like object in MPI. If you
>>>>> protect writes to the file with an exclusive lock you can serialize the
>>>>> writes and make use of NFS's close to open cache coherence.
>>>>>
>>>>> nick
>>>>>
>>>>>
>>>>> On Tue, Feb 2, 2010 at 08:27, Laurence Marks<L-marks_at_[hidden]>
>>>>> wrote:
>>>>> I have a question concerning having many processors in a mpi job all
>>>>> write to the same file -- not using mpi calls but with standard
>>>>> fortran I/O. I know that this can lead to consistency issues, but it
>>>>> can also lead to OS issues with some flavors of nfs.
>>>>>
>>>>> At least in fortran, there is nothing "wrong" with doing this. My
>>>>> question is whether this is "One of the Seven Deadly Sins" of mpi
>>>>> programming, or just frowned on. (That is, it should be OK even if it
>>>>> leads to nonsense files, and not lead to OS issues.) If it is a sin, I
>>>>> would appreciate a link to where this is spelt out in some "official"
>>>>> document or similar.
>>>>>
>>>>> --
>>>>> Laurence Marks
>>>>> Department of Materials Science and Engineering
>>>>> MSE Rm 2036 Cook Hall
>>>>> 2220 N Campus Drive
>>>>> Northwestern University
>>>>> Evanston, IL 60208, USA
>>>>> Tel: (847) 491-3996 Fax: (847) 491-7820
>>>>> email: L-marks at northwestern dot edu
>>>>> Web: www.numis.northwestern.edu
>>>>> Chair, Commission on Electron Crystallography of IUCR
>>>>> www.numis.northwestern.edu/
>>>>> Electron crystallography is the branch of science that uses electron
>>>>> scattering and imaging to study the structure of matter.
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>>
>>>> --
>>>> Jeff Squyres
>>>> jsquyres_at_[hidden]
>>>>
>>>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
>
>