On Wed, 21 May 2008, Jeff Squyres wrote:
> 2. An out-of-the-box "mpirun a.out" will print warning messages in
> perfectly valid/good configurations (no verbs-capable hardware, but
> just happen to have libibverbs installed). This is a Big Deal.
Which is easily solved with a better error message, as Pasha suggested.
> 3. Problems with HCA hardware and/or verbs stack are uncommon
> (nowadays). I'd be ok asking someone to enable a debug flag to get
> more information on configuration problems or hardware faults.
> Shouldn't we be optimizing for the common case?
> In short: I think it's no longer safe to assume that machines with
> libibverbs installed must also have verbs-capable hardware.
But here's the real problem -- with our current selection logic, a user
with libibverbs but no IB cards gets an error message saying "hey, we need
you to set this flag to make this error go away" (or would, per Pasha's
suggestion). A user with a busted IB stack on a node (which we still saw
pretty often at LANL) starts using TCP and their application runs like a
I guess it's a matter of how often you see errors in the IB stack that
cause nic initialization to fail. The machines I tend to use still
exhibit this problem pretty often, but it's possible I just work on bad
hardware more often than is usual in the wild.
It would be great if libibverbs could return two different error messages
- one for "there's no IB card in this machine" and one for "there's an IB
card here, but we can't initialize it". I think that would make this
argument go away. Open MPI could probably mimic that behavior by parsing
the PCI tables, but that sounds ... painful.
I guess the root of my concern is that unexpected behavior with no
explanation is (in my mind) the most dangerous case and the one we should
address by default. And turning this error message off is going to cause
unexpected behavior without explanation.
Just my $0.02.