Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] poor performance using the openib btl
From: Fischer, Greg A. (fischega_at_[hidden])
Date: 2014-06-25 10:46:34

Attached are the results of "grep thread" on my configure output. There appears to be some amount of threading, but is there anything I should look for in particular?

I see Mike Dubman's questions on the mailing list website, but his message didn't appear to make it to my inbox. The answers to his questions are:

[binford:fischega] $ rpm -qa | grep ofed

Distro: SLES11 SP3

[binf102:fischega] $ /usr/sbin/ibstat
CA 'mlx4_0'
        CA type: MT26428

Command line (path and LD_LIBRARY_PATH are set correctly):
mpirun -x LD_LIBRARY_PATH -mca btl openib,sm,self -mca btl_openib_verbose 1 -np 31 $CTF_EXEC

From: users [mailto:users-bounces_at_[hidden]] On Behalf Of Maxime Boissonneault
Sent: Tuesday, June 24, 2014 6:41 PM
To: Open MPI Users
Subject: Re: [OMPI users] poor performance using the openib btl

What are your threading options for OpenMPI (when it was built) ?

I have seen OpenIB BTL completely lock when some level of threading is enabled before.

Maxime Boissonneault

Le 2014-06-24 18:18, Fischer, Greg A. a écrit :
Hello openmpi-users,

A few weeks ago, I posted to the list about difficulties I was having getting openib to work with Torque (see "openib segfaults with Torque", June 6, 2014). The issues were related to Torque imposing restrictive limits on locked memory, and have since been resolved.

However, now that I've had some time to test the applications, I'm seeing abysmal performance over the openib layer. Applications run with the tcp btl execute about 10x faster than with the openib btl. Clearly something still isn't quite right.

I tried running with "-mca btl_openib_verbose 1", but didn't see anything resembling a smoking gun. How should I go about determining the source of the problem? (This uses the same OpenMPI Version 1.8.1 / SLES11 SP3 / GCC 4.8.3 setup discussed previously.)



users mailing list



Link to this post:

Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Ph. D. en physique