Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Parallel I/O with MPI-1
From: Robert Latham (robl_at_[hidden])
Date: 2008-07-24 09:25:59

On Wed, Jul 23, 2008 at 09:47:56AM -0400, Robert Kubrick wrote:
> HDF5 supports parallel I/O through MPI-I/O. I've never used it, but I
> think the API is easier than direct MPI-I/O, maybe even easier than raw
> read/writes given its support for hierarchal objects and metadata.

In addition to the API provided by parallel HDF5 and parallel-NetCDF,
these high level libraries offer a self-describing portable file
format. Pretty nice when collaborating with others. Plus there are a
host of viewers for these file formats, so that's another thing you
don't have to worry about.

> HDF5 supports multiple storage models and it supports MPI-IO.
> HDF5 has an open interface to access raw storage. This enables HDF5
> files to be written to a variety of media, including sequential files,
> families of files, memory, Unix sockets (i.e., a network).
> New "Virtual File" drivers can be added to support new storage access
> mechanisms.
> HDF5 also supports MPI-IO with Parallel HDF5. When building HDF5,
> parallel support is included by configuring with the --enable-parallel
> option. A tutorial for Parallel HDF5 is included with the HDF5 Tutorial
> at:
> /HDF5/Tutor/

It's a very good tutorial. Do read the parallel I/O chapter closely,
especially the parts about enabling collective I/O via property lists
and transfer templates. For many HDF5 workloads today, collective I/O
is the key to getting good performance (this was not always the case
back in the bad old days of MPICH1 and LAM, but has been since at
least the HDF5-1.6 series).


Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Lab, IL USA                 B29D F333 664A 4280 315B