Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] OpenIB problems
From: Ogden, Jeffry Brandon (jbogden_at_[hidden])
Date: 2007-11-28 12:18:19


For what it's worth Andrew, the RETRY_EXCEEDED_ERRORS can be caused by
flaky hardware as well. The timeout value is probably best tuned
relative to the size of your IB fabric. But if reliability is the
biggest criteria, crank up the timemout value to 20. That's the best
you can do. If it continues to happen, it is more than likely you have
a flaky HCA, IB link, switch side sw, or node. We actually have way too
much IB hardware for any sane person and my experience is that the
RETRY_EXCEEDED_ERRORS can sometimes be really tricky to track down. One
of my favorites is the spontaneous rebooting node. We see nodes under
heavy MPI application load sometimes randomly reboot. This causes the
RETRY_EXCEEDED_ERROR as well. I would second the recommendation to
watch the IB counters across the entire IB fabric from the subnet
manager.

Good luck!

> -----Original Message-----
> From: users-bounces_at_[hidden]
> [mailto:users-bounces_at_[hidden]] On Behalf Of Andrew Friedley
> Sent: Wednesday, November 28, 2007 9:36 AM
> To: Open MPI Users
> Subject: Re: [OMPI users] OpenIB problems
>
> What value do you suggest then? I know I've seen the problem
> persist at
> values of 14 and 16, and would rather be certain that this
> isn't going
> to kill the job that just sat in the queue for a week.
>
> Andrew
>
> Jeff Squyres wrote:
> > Roland thought that the default value of 10 might be a bit
> too low and
> > that tuning it to be higher, particularly in apps that pound on a
> > single port, would probably be acceptable.
> >
> > Tuning up to 20 is probably a bit overkill.
> >
> >