Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Openmpi and processor affinity
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2009-06-02 21:25:54

On Jun 2, 2009, at 7:30 PM, Iftikhar Rathore -X (irathore - SIFY
LIMITED at Cisco) wrote:

> We are using openmpi version 1.2.8 (packaged with ofed-1.4). I am
> trying
> to run hpl-2.0 (linpak). We have two intel quad core CPU's in all our
> server (8 total cores) and all hosts in the hostfile have lines that
> look like " slots=8max_slots=8".

Per Gus's mail, let's assume that this is a typo in email.

> Now when I use mpirun (even with --mca mpi_paffinity_alone 1) it does
> not keep the affinity, the processes seem to gravitate towards first
> four cores (using top and hitting 1). I know I do have MCA paffinity
> available.
> [root_at_devi DLR_WB_88]# ompi_info | grep paffinity
> [] mca: base: component_find: unable to open btl
> openib: file not found (ignored)

The above line is worrysome -- it means that it is unable to open the
openib BTL, which means that you're not using the OpenFabrics
networking. You should be able to run ompi_info and see that it lists
the openib BTL properly.

This *usually* means that library dependencies are missing (or perhaps
you are accidentally mixing-n-matching multiple versions of Open MPI
on one or more boxes?). Run ld on $prefix/lib/openmpi/ and see what libraries are missing.

This is probably a more serious performance issue to fix than affinity.

> MCA paffinity: linux (MCA v1.0, API v1.0, Component v1.2.8)
> The command line I am using is:
> # mpirun -nolocal -np 896 -v --mca mpi_paffinity_alone 1 -hostfile /
> mnt/apps/hosts/896_8slots /mnt/apps/bin/xhpl

There's no reason that HPL should override paffinity. Drew (another
TME from Cisco) tells me in an off-list email that you're using top to
verify that the affinity is changing. Try using htop (
) to verify. My old RHEL4-based machines' version of top don't show
affinity -- perhaps newer versions do. But I know htop does.

Jeff Squyres
Cisco Systems