If you set the eager size large enough, the isend will not return
until the data is pushed into the network layer. However, this doesn't
guarantee that the data is delivered to the peer, but only that it was
queued in the network (in the TCP case it is copied somewhere in the
kernel buffers). The kernel will deliver that data doing a best
effort, but there is no guarantee on that. As the kernel buffer has a
limited size (no more than 128k) the expected graphical behavior for
the isend operation over TCP should look like stairs, slightly going
up because of the memcpy, and a large jump for every syscall required
to do the operation.
Now for the irecv the story is a lot more complex. The irecv only
realize the matching, and if the data is not yet completely available
(let's say only a small fragment was received at the moment of the
irecv), the irecv will return (there is no eager notion there). The
remaining of the data will became available only after the
On Dec 8, 2008, at 12:50 , Barry Rountree wrote:
> ---- Original message ----
>> Date: Mon, 8 Dec 2008 11:47:19 -0500
>> From: George Bosilca <bosilca_at_[hidden]>
>> Subject: Re: [OMPI users] How to force eager behavior during Isend?
>> To: Open MPI Users <users_at_[hidden]>
>> These values are used deep inside the Open MPI library, in order to
>> define how we handle the messages internally. From a user perspective
>> you will not see much difference.
> Well, this user does.... ;-) I'm working on how to slow the
> processor when the library is blocked, and it's my understanding
> from the state-of-the-openmpi-union talk at SC that blocking
> progress functionality wouldn't be implemented for a while yet. So
> I'm making do with home-rolled solutions, one of which is figuring
> out the expected time needed to transfer a certain amount of data,
> then assuming any time spent in the library in excess of that is
> blocking time.
> The fact that the behavior of Isend and Irecv is determined by
> message size makes this a little more difficult, so I wanted to be
> able to either force eager sending in nearly all cases or turn off
> eager sending entirely. Once the algorithm is working in these
> simpler cases I can go back and make it work for the optimized case.
>> Moreover, checking the number of
>> completed requests returned by MPI_Wait* if definitively not the most
>> accurate way to test this.
> I'm not sure what you mean here. My PMPI library uses timestamps to
> determine how long was spent inside and outside the library. If the
> Isends are eager, very little time should be spent in the
> corresponding Wait, correct?
>> If you really want to see that these
>> parameters are taken in account you will need to a lower level
>> interface, such as PERUSE.
> Ok, that looks like it could be useful. Google has your
> "Implementation and Usage" paper as the first hit; I'll read p on
> users mailing list