Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] openmpi credits for eager messages
From: Brian W. Barrett (brbarret_at_[hidden])
Date: 2008-02-04 15:16:30


Now that this discussion has gone way off into the MPI standard woods :).

Was your test using Open MPI 1.2.4 or 1.2.5 (the one with the segfault)?
There was definitely a bug in 1.2.4 that could cause exactly the behavior
you are describing when using the shared memory BTL, due to a silly
delayed initialization bug/optimization.

If you are using the OB1 PML (the default), you will still have the
possibility of running the receiver out of memory if the unexpected queue
grows without bounds. I'll withold my opinion on what the standard says
so that we can perhaps actually help you solve your problem and stay out
of the weeds :). Note however, that in general unexpected messages are a
bad idea and thousands of them from one peer to another should be avoided
at all costs -- this is just good MPI programming practice.

Now, if you are using MX, you can replicate MPICH/MX's behavior (including
the very slow part) by using the CM PML (--mca pml cm on the mpirun
command line), which will use the MX library message matching and
unexpected queue and therefore behave exactly like MPICH/MX.

Brian

On Sat, 2 Feb 2008, 8mj6tc902_at_[hidden] wrote:

> That would make sense. I able to break OpenMPI by having Node A wait for
> messages from Node B. Node B is in fact sleeping while Node C bombards
> Node A with a few thousand messages. After a while Node B wakes up and
> sends Node A the message it's been waiting on, but Node A has long since
> been buried and seg faults. If I decrease the number of messages C is
> sending, it works properly. This was on OpenMPI 1.2.4 (using I think the
> SM BTL (might have been MX or TCP, but certainly not infiniband. I could
> dig up the test and try again if anyone is seriously curious).
>
> Trying the same test on MPICH/MX went very very slow (I don't think they
> have any clever buffer management) but it didn't crash.
>
> Sacerdoti, Federico Federico.Sacerdoti-at-deshaw.com
> |openmpi-users/Allow| wrote:
>> Hi,
>>
>> I am readying an openmpi 1.2.5 software stack for use with a
>> many-thousand core cluster. I have a question about sending small
>> messages that I hope can be answered on this list.
>>
>> I was under the impression that if node A wants to send a small MPI
>> message to node B, it must have a credit to do so. The credit assures A
>> that B has enough buffer space to accept the message. Credits are
>> required by the mpi layer regardless of the BTL transport layer used.
>>
>> I have been told by a Voltaire tech that this is not so, the credits are
>> used by the infiniband transport layer to reliably send a message, and
>> is not an openmpi feature.
>>
>> Thanks,
>> Federico
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>