Thanks for the quick reply and suggestions.<br><br>I have tried both isolating the output to a single OST and striping
across multiple OSTs. Both will produce the same result. I have tried
compiling with multiple versions of both pathscale and intel compilers
all with the same result.<br>
<br>The odd thing is that this seems to work using hpmpi 2.03 compiled with
pathscale 3.2 and intel 10.1.018. The operating system is XC 3.2.1
which is essentially rhel4.5. The kernel is 2.6.9-67.9hp.7sp.XCsmp.
Lustre version is lustre-1.4.11-2.3_0.6_xc3.2.1_<div id=":40" class="ii gt">k2.6.9_67.9hp.7sp.XCsmp.<br><br>Thanks for the info, Nate<br></div><br><br><div class="gmail_quote">On Tue, Mar 3, 2009 at 11:10 AM, Brian Dobbins <span dir="ltr">&lt;<a href="mailto:bdobbins@gmail.com">bdobbins@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>Hi Nathan,<br><br>  I just ran your code here and it worked fine - CentOS 5 on dual Xeons w/ IB network, and the kernel is 2.6.18-53.1.14.el5_lustre.1.6.5smp.  I used an OpenMPI 1.3.0 install compiled with Intel 11.0.081 and, independently, one with GCC 4.1.2.  I tried a few different times with varying numbers of processors.  <br>

<br>  (Both executables were compiled with -O2)<br><br>  I&#39;m sure the main OpenMPI guys will have better ideas, but in the meantime what kernel, OS and compilers are you using?  And does it happen when you write to a single OST?  Make a directory and try setting the stripe-size to 1 (eg, lfs setstripe &lt;directory name&gt; 1048576 0 1&#39; will give you, I think, a 1MB stripe size starting at OST 0 and of size 1.)  I&#39;m just wondering whether it&#39;s something with your hardware, maybe a particular OST, since it seems to work for me.<br>

<br>  ... Sorry I can&#39;t be of more help, but I imagine the regular experts will chime in shortly.<br><br>  Cheers,<br>  - Brian<br><br><br><div class="gmail_quote"><div><div></div><div class="h5">On Tue, Mar 3, 2009 at 12:51 PM, Nathan Baca <span dir="ltr">&lt;<a href="mailto:nathan.baca@gmail.com" target="_blank">nathan.baca@gmail.com</a>&gt;</span> wrote:<br>

</div></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div></div><div class="h5">Hello,<br><br>I am seeing inconsistent mpi-io behavior when writing to a Lustre file system using open mpi 1.3 with romio. What follows is a simple reproducer and output. Essentially one or more of the running processes does not read or write the correct amount of data to its part of a file residing on a Lustre (parallel) file system.<br>


<br>Any help figuring out what is happening is greatly appreciated. Thanks, Nate<br><br>program gcrm_test_io
<br>  implicit none
<br>  include &quot;mpif.h&quot;
<br> <br>  integer X_SIZE
<br> <br>      integer w_me, w_nprocs
<br>      integer  my_info
<br> <br>      integer i
<br>      integer (kind=4) :: ierr
<br>      integer (kind=4) :: fileID
<br>      
<br>      integer (kind=MPI_OFFSET_KIND)        :: mylen
<br>      integer (kind=MPI_OFFSET_KIND)        :: offset
<br>      integer status(MPI_STATUS_SIZE)
<br>      integer count
<br>      integer ncells
<br>      real (kind=4), allocatable, dimension (:)     :: array2
<br>      logical sync
<br> <br>      call mpi_init(ierr)
<br>      call MPI_COMM_SIZE(MPI_COMM_WORLD,w_nprocs,ierr)
<br>      call MPI_COMM_RANK(MPI_COMM_WORLD,w_me,ierr)
<br> <br>      call mpi_info_create(my_info, ierr)
<br>!     optional ways to set things in mpi-io
<br>!     call mpi_info_set   (my_info, &quot;romio_ds_read&quot; , &quot;enable&quot;   , ierr)
<br>!     call mpi_info_set   (my_info, &quot;romio_ds_write&quot;, &quot;enable&quot;   , ierr)
<br>!     call mpi_info_set   (my_info, &quot;romio_cb_write&quot;, &quot;enable&quot;    , ierr)
<br> <br>      x_size = 410011  ! A &#39;big&#39; number, with bigger numbers it is more likely to fail
<br>      sync = .true.  ! Extra file synchronization
<br> <br>      ncells = (X_SIZE * w_nprocs)
<br> <br>!  Use node zero to fill it with nines
<br>      if (w_me .eq. 0) then
<br>          call MPI_FILE_OPEN  (MPI_COMM_SELF, &quot;output.dat&quot;, MPI_MODE_CREATE+MPI_MODE_WRONLY, my_info, fileID, ierr)
<br>          allocate (array2(ncells)) 
<br>          array2(:) = 9.0
<br>          mylen = ncells
<br>          offset = 0 * 4
<br>          call MPI_FILE_SET_VIEW(fileID,offset, MPI_REAL,MPI_REAL, &quot;native&quot;,MPI_INFO_NULL,ierr)
<br>          call MPI_File_write(fileID, array2, mylen , MPI_REAL, status,ierr)
<br>          call MPI_Get_count(status,MPI_INTEGER, count, ierr)
<br>          if (count .ne. mylen) print*, &quot;Wrong initial write count:&quot;, count,mylen
<br>          deallocate(array2)
<br>          if (sync) call MPI_FILE_SYNC (fileID,ierr)
<br>          call MPI_FILE_CLOSE (fileID,ierr)
<br>      endif
<br> <br>!  All nodes now fill their area with ones
<br>      call MPI_BARRIER(MPI_COMM_WORLD,ierr)
<br>      allocate (array2( X_SIZE))
<br>      array2(:) = 1.0
<br>      offset = (w_me * X_SIZE) * 4 ! multiply by four, since it is real*4
<br>      mylen = X_SIZE
<br>      call MPI_FILE_OPEN  (MPI_COMM_WORLD,&quot;output.dat&quot;,MPI_MODE_WRONLY, my_info, fileID, ierr)
<br>      print*,&quot;node&quot;,w_me,&quot;starting&quot;,(offset/4) + 1,&quot;ending&quot;,(offset/4)+mylen
<br>      call MPI_FILE_SET_VIEW(fileID,offset, MPI_REAL,MPI_REAL, &quot;native&quot;,MPI_INFO_NULL,ierr)
<br>      call MPI_File_write(fileID, array2, mylen , MPI_REAL, status,ierr)
<br>      call MPI_Get_count(status,MPI_INTEGER, count, ierr)
<br>      if (count .ne. mylen) print*, &quot;Wrong write count:&quot;, count,mylen,w_me
<br>      deallocate(array2)
<br>      if (sync) call MPI_FILE_SYNC (fileID,ierr)
<br>      call MPI_FILE_CLOSE (fileID,ierr)
<br> <br>!  Read it back on node zero to see if it is ok data
<br>      if (w_me .eq. 0) then
<br>          call MPI_FILE_OPEN  (MPI_COMM_SELF, &quot;output.dat&quot;, MPI_MODE_RDONLY, my_info, fileID, ierr)
<br>          mylen = ncells
<br>          allocate (array2(ncells))
<br>          call MPI_File_read(fileID, array2, mylen , MPI_REAL, status,ierr)
<br>          call MPI_Get_count(status,MPI_INTEGER, count, ierr)
<br>          if (count .ne. mylen) print*, &quot;Wrong read count:&quot;, count,mylen
<br>          do i=1,ncells
<br>               if (array2(i) .ne. 1) then
<br>                  print*, &quot;ERROR&quot;, i,array2(i), ((i-1)*4), ((i-1)*4)/(1024d0*1024d0) ! Index, value, # of good bytes,MB
<br>                  goto 999
<br>               end if
<br>          end do
<br>          print*, &quot;All done with nothing wrong&quot;
<br> 999      deallocate(array2)
<br>          call MPI_FILE_CLOSE (fileID,ierr)
<br>          call MPI_file_delete (&quot;output.dat&quot;,MPI_INFO_NULL,ierr)
<br>      endif
<br> <br>      call mpi_finalize(ierr)
<br> <br>end program gcrm_test_io  
<br> <br clear="all">1.3 Open MPI<br> node           0 starting                     1 ending                410011<br> node           1 starting                410012 ending                820022<br> node           2 starting                820023 ending               1230033<br>


 node           3 starting               1230034 ending               1640044<br> node           4 starting               1640045 ending               2050055<br> node           5 starting               2050056 ending               2460066<br>


 All done with nothing wrong<br><br><br> node           0 starting                     1 ending                410011<br> node           1 starting                410012 ending                820022<br> node           2 starting                820023 ending               1230033<br>


 node           5 starting               2050056 ending               2460066<br> node           4 starting               1640045 ending               2050055<br> node           3 starting               1230034 ending               1640044<br>


 Wrong write count:      228554                410011           2<br> Wrong read count:     1048576               2460066<br> ERROR     1048577  0.0000000E+00     4194304   4.00000000000000    <br><br><br> node           1 starting                410012 ending                820022<br>


 node           0 starting                     1 ending                410011<br> node           2 starting                820023 ending               1230033<br> node           3 starting               1230034 ending               1640044<br>


 node           4 starting               1640045 ending               2050055<br> node           5 starting               2050056 ending               2460066<br> Wrong read count:     1229824               2460066<br> ERROR     1229825  0.0000000E+00     4919296   4.69140625000000<br>

<font color="#888888">
<br>-- <br>Nathan Baca<br><a href="mailto:nathan.baca@gmail.com" target="_blank">nathan.baca@gmail.com</a><br>
</font><br></div></div>_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@open-mpi.org" target="_blank">users@open-mpi.org</a><br>
<a href="http://www.open-mpi.org/mailman/listinfo.cgi/users" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/users</a><br></blockquote></div><br>
<br>_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@open-mpi.org">users@open-mpi.org</a><br>
<a href="http://www.open-mpi.org/mailman/listinfo.cgi/users" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/users</a><br></blockquote></div><br><br clear="all"><br>-- <br>Nathan Baca<br><a href="mailto:nathan.baca@gmail.com">nathan.baca@gmail.com</a><br>