Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] sending/receiving large buffers
From: Martin Siegert (siegert_at_[hidden])
Date: 2009-11-09 14:57:36


Hi Vincent,

On Mon, Nov 09, 2009 at 11:45:29AM +0100, Vincent Loechner wrote:
>
> Martin,
>
> > I expect problems with sizes larger than 2^31-1, but these array sizes
> > are still much smaller.
> No, they are bigger, you allocate two arrays of 320 Mdouble :
> 2 * 320M * 8 = 5GB.
>
> Are your processes limited to 4GB virtual memory ?

No, they aren't. There are 16GB on each node plus swap space; I have
run processes much bigger than this - in fact this program doesn't
even come close to the memory limits on each node.

The reason why there is a problem for counts larger than 2^31-1 is
that the "count" variable in the MPI library is a MPI_INT - a 32bit integer.

BTW: the MPI_Allreduce version fails (i.e., hangs forever) for an
array size larger than 176763240. This is not a smooth transition:
for 176763240 the program finishes in about 3s, for 176763241 it
just hangs.
This looks to me as if there is an internal buffer or some other
limit that I cannot exceed. Is there such a limit? Or: can somebody
confirm that they can run the program with an array size larger
than 176763240?

Cheers,
Martin