Hello All,

I have just rebuilt openmpi-1.4-3 on our cluster, and I see this error:

It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  orte_grpcomm_modex failed
  --> Returned "Data unpack would read past end of buffer" (-26) instead of "Success" (0)
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 25633 on node tik40x exited on signal 27 (Profiling timer expired).


[tik40x:25626] [[29400,0],0] odls:default:fork binding child [[29400,1],0] to slot_list 0:0
[tik40x:25633] [[29400,1],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file grpcomm_bad_module.c at line 535
*** The MPI_Init() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[tik40x:25633] Abort before MPI_INIT completed successfully; not able to guarantee that all other processes were killed!


I had already tested this application prior to rebuilding openmpi (same version, but without thread support), and it was running well.

I have some discussions on this error in the forum, but I am not getting any useful pointers.

Has anyone else also seen this error?

Best

Devendra Rai


From: German Hoecht <german.hoecht@googlemail.com>
To: Open MPI Users <users@open-mpi.org>; Rob Latham <robl@mcs.anl.gov>
Sent: Wednesday, 28 September 2011, 10:09
Subject: Re: [OMPI users] maximum size for read buffer in MPI_File_read/write

Hi Rob,

thanks for your comments. I understand that it's most probably not worth
the effort to find the actual reason.

Because I have to deal with very large files I preferred using
"std::numeric_limits<int>::max()" rather than a hard-coded value
to split the read in case an IO request exceeds this amount. (This is
not the usual case but can happen.)

So your advice to use a max IO buffer of 1GB is quite precious.

To be honest, I did not do the check before we observed strange
numbers... Usually, MPI/ROMIO read/write functions are very stable, the
concerned code has read several Terabytes in the meanwhile.

Best regards,
German

On 09/27/2011 10:01 PM, Rob Latham wrote:
> On Thu, Sep 22, 2011 at 11:37:10PM +0200, German Hoecht wrote:
>> Hello,
>>
>> MPI_File_read/write functions uses  an integer to specify the size of
>> the buffer, for instance:
>> int MPI_File_read(MPI_File fh, void *buf, int count, MPI_Datatype
>> datatype, MPI_Status *status)
>> with:
>> count    Number of elements in buffer (integer).
>> datatype  Data type of each buffer element (handle).
>>
>> However, using the maximum value of 32 bytes integers:
>> count = 2^31-1 = 2147483647 (and datatype = MPI_BYTE)
>> MPI_file_read only reads  2^31-2^12 = 2147479552 bytes.
>> This means that 4095 bytes are ignored.
>>
>> I am not aware of this specific limit for integers in (Open) MPI
>> function calls. Is this supposed to be correct?
>
> Hi.  I'm the ROMIO maintainer.  OpenMPI more or less rolls up ROMIO
> into OpenMPI, so any problems with the MPI_File_* routines is in my
> lap, not OpenMPI.
>
> I'll be honest with you: i've not given any thought to just how big
> the biggest request could be.  The independent routines, especially
> with a simple type like MPI_BYTE, are going to almost immediately call
> the underlying posix read() or write() call.
>
> I can confirm the behavior you observe with your test program.
> Thanks much for providing one.  I'll dig around but I cannot think of
> something in ROMIO that would ignore these 4095 bytes.  I do think
> it's legal by the letter of the standard to read or write less than
> requested.  "Upon completion, the amount of data accessed by the
> calling process is returned in a status." 
>
> Bravo to you for actually checking return values and the status.  I
> don't think many non-library codes do that :>
>
> I should at least be able to explain the behavior, so I'll dig a bit.
>
> in general, if you plot "i/o performance vs blocksize", every file
> system tops out around several tens of megabytes.  So, we have given
> the advice to just split up this nearly 2 gb request into several 1 gb
> requests. 
>
> ==rob
>

_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users