Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

From: Edgar Gabriel (gabriel_at_[hidden])
Date: 2005-07-18 10:08:06

We have currently Barbara Chapman from the University of Houston as a
guest scientist here at Stuttgart. Most of you might now, that Barbara
is working on compiler design and OpenMP issues. This afternoon she
dropped in my office and asked me, whether the Open MPI group has
thought/discussed processor affinity issues up to now (which we just did
:-) ).

Anyway, I just wanted to point out, that various people from the OpenMP
community have been working/are still working on this issue, and that it
might be interesting to exchange information and maybe coordinate the
approaches. I cc'ed Barbara therefore also on this email...


Rich L. Graham wrote:
> On Jul 18, 2005, at 6:28 AM, Jeff Squyres wrote:
>>On Jul 18, 2005, at 2:50 AM, Matt Leininger wrote:
>>>>Generally speaking, if you launch <=N processes in a job on a node
>>>>(where N == number of CPUs on that node), then we set processor
>>>>affinity. We set each process's affinity to the CPU number according
>>>>to the VPID ordering of the procs in that job on that node. So if
>>>>launch VPIDs 5, 6, 7, 8 on a node, 5 would go to processor 0, 6 would
>>>>go to processor 1, etc. (it's an easy, locally-determined ordering).
>>> You'd need to be careful with dual-core cpus. Say you launch a 4
>>>task MPI job on a 4-socket dual core Opteron. You'd want to schedule
>>>the tasks on nodes 0, 2, 4, 6 - not 0, 1, 2, 3 to get maximum memory
>>>bandwidth to each MPI task.
>>With the potential for non-trivial logic like this, perhaps the extra
>>work for a real framework would be justified, then.
>>> Also, how would this work with hybrid MPI+threading (either
>>>or OpenMP) applications? Let's say you have an 8 or 16 cpu node and
>>>start up 2 MPI tasks with 4 compute threads in each task. The optimum
>>>layout may not be running the MPI tasks on cpu's 0 and 1. Several
>>>hybrid applications that ran on ASC White and now Purple will have
>>Hum. Good question. The MPI API doesn't really address this -- the
>>MPI API is not aware of additional threads that are created until you
>>call an MPI function (and even then, we're not currently checking which
>>thread is calling -- that would just add latency).
>>What do these applications do right now? Do they set their own
>>processor / memory affinity? This might actually be outside the scope
>>of MPI...? (I'mm not trying to shrug off responsibility, but this
>>might be a case where the MPI simply doesn't have enough information,
>>and to get that information [e.g., via MPI attributes or MPI info
>>arguments] would be more hassle than the user just setting the affinity
> If you set things up such that you can specify input parameters on where
> to put each process, you have the flexibility you want. The locality
> API's
> I have seen all mimiced the IRIX API, which had these capabilities. If
> you
> want some ideas, look at LA-MPI, it does this - the implementation is
> pretty strange (just he coding), but it is there.
> Rich
>>{+} Jeff Squyres
>>{+} The Open MPI Project
>>devel mailing list
> _______________________________________________
> devel mailing list
> devel_at_[hidden]

Dr.-Ing. Edgar Gabriel
Clusters and Distributed Units
High Performance Computing Center Stuttgart (HLRS)
University of Stuttgart
Tel: +49 711 685 8039      
Fax: +49 711 678 7626                e-mail:gabriel_at_[hidden]