Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Mellanox MLX4_EVENT_TYPE_SRQ_LIMIT kernel messages
From: Dave Love (d.love_at_[hidden])
Date: 2012-10-04 10:40:51


Meanwhile, much later -- you'll sympathize: Did you have any joy with
this?

You wrote:

> These messages appeared when running IMB compiled with openmpi 1.6.1
> across 256 cores (16 nodes, 16 cores per node). The job ran from
> 09:56:54 to 10:08:46 and failed with no obvious error messages.

I don't know about the messages, but there are successful ×256 runs in
my ~/imb, one with default params, and also ×512. The 256 ones were
accounted ∼330GB, and the default h_vmem is still 1G. That's not the
cause of the failure, is it?

For the kernel issue, do you actually have the same adaptors under RH5
to compare? lspci says our current QDR ones are the same as yours
(surprisingly), and they're OK with openib params from the default
Mellanox OFED setup. They're on older OFED (not vanilla RH) due to our
horrible hardware mixture.

With the new stuff I can't do more than access a head. That also has
the same adaptors, but I don't know if MPI runs have your symptom. The
modprobe config is different from yours, but the driver is older. If it
might help, I could poke around node images and send config files, but I
don't know what the various different images are.

I might have hoped you wouldn't have to sort this yourself, if I was a
beginner...