Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] New Romio for OpenMPI available in bitbucket
From: Pascal Deveze (Pascal.Deveze_at_[hidden])
Date: 2010-09-17 06:36:00


Hi all,

In charge of ticket 1888 (see at
https://svn.open-mpi.org/trac/ompi/ticket/1888) ,
I have put the resulting code in bitbucket at:
http://bitbucket.org/devezep/new-romio-for-openmpi/

The work in this repo consisted in refreshing ROMIO to a newer
version: the one from the very last MPICH2 release (mpich2-1.3b1).

Testing:
  1. runs fine except one minor error (see the explanation below) on
various FS.
  2. runs fine with Lustre, but:
     . had to add a small patch in romio/adio/ad_lustre_open.c
  3. see below how to efficiently run with Lustre.

You are invited to test and send comments

Enjoy !

Pascal

======== The minor error ===================
The test error.c fails because OpenMPI does not handle correctly the
"two level" error functions of ROMIO:
    error_code = MPIO_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE,
                                        myname, __LINE__, MPI_ERR_ARG,
                                        "**iobaddisp", 0);
OpenMPI limits its view to MPI_ERR_ARG, but the real error is
"**iobaddisp".

========= How to test performances with Lustre ===================
1) Compile with Lustre ADIO driver. For this, add the flag
        --with-io-romio-flags="--with-file-system=ufs+nfs+lustre" to
your configure command.

2) Of course, you should have a Lustre file system mounted on all the
nodes you will run on.

3) Take an application like coll_perf.c (in the test directory). In this
application, change the
    three dimensions to 1000, that will create a file of 4 GB (big
files are required in order
    to reach good performances with Lustre).

4) Put the highest possible striping_factor in the hint. For this, one
solution is :
 - If your Lustre file system have 16 OST, create a hint file with the
following line:
    striping_factor 16
- Export the path to this file in the variable ROMIO_HINTS:
   export ROMIO_HINTS=my_directory/my_hints
 If you do not specify the striping_factor, Lustre will set the default
value (often 2 only).
 You can verify the striping_factor set by Lustre with the following
command:
     lfs getstripe <file> (look at the value of lmm_stripe_count)
  Note: The striping_factor is set once at file creation and cannot be
changed after.

5) Run your test, specifying a file located in the Lustre file system.