Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2005-08-25 05:57:47


Galen --

When you guys return to the office, it's probably worthwhile to see if
it's possible to set leave_pinned to 1 automatically if the memory
hooks are being set properly (Brian keeps explaining this stuff to me,
but it's so complicated that the details quickly fall out of my brain
;-) ).

On Aug 25, 2005, at 12:48 AM, Galen M. Shipman wrote:

> Hi Troy,
>
> Tim and I would like to discuss this with you as well. One thing I
> would ask, are you using the btl_mvapi_leave_pinned=1 option?
> otherwise it is not a apples to apples comparison.
>
> - Galen
> On Aug 24, 2005, at 8:21 PM, Troy Benjegerdes wrote:
>
>> I have some Netpipe graphs of OpenMPI and Mviapich on OpenIB gen2 on
>> Opteron systems, one with PCI-X IB cards, and the other with PCI-
>> Express
>> DDR IB cards.
>>
>> I'd like to chat with someone who fill me in a bit on what's going on
>> with performance, and how the BTL for IB is implemented. One thing I'd
>> like to try is modifying the kernel to allow 95% of physical memory to
>> be registered for IB, and see how much of a difference this makes.
>>
>> I am also toying with the idea of writing an OpenIB BTL module for
>> Netpipe, so I can evaluate the BTL layer performance directly without
>> any MPI overhead.
>>
>> I also get the following messages on startup:
>>
>> libibverbs: Warning: no userspace device-specific driver found for
>> uverbs0 driver search path: /usr/lib/infiniband
>> [0,1,1][btl_openib_component.c:267:mca_btl_openib_component_init]
>> No hca's found on this host!
>>
>> Something strange is going on, because it is finding at least 1 IB
>> port, given the performance numbers.

-- 
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/