Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

From: Richard L. Graham (rlgraham_at_[hidden])
Date: 2005-07-18 10:26:43

To add to this, I would suggest not coupling processor affinity and
memory locality at the component level. At some level you do need
to tie these together, but there are other components that also need
to be considered, such as NIC locality, and probably other things
too ...


On Jul 18, 2005, at 8:50 AM, Matt Leininger wrote:

> On Mon, 2005-07-18 at 08:28 -0400, Jeff Squyres wrote:
>> On Jul 18, 2005, at 2:50 AM, Matt Leininger wrote:
>>>> Generally speaking, if you launch <=N processes in a job on a node
>>>> (where N == number of CPUs on that node), then we set processor
>>>> affinity. We set each process's affinity to the CPU number
>>>> according
>>>> to the VPID ordering of the procs in that job on that node. So if
>>>> you
>>>> launch VPIDs 5, 6, 7, 8 on a node, 5 would go to processor 0, 6
>>>> would
>>>> go to processor 1, etc. (it's an easy, locally-determined ordering).
>>> You'd need to be careful with dual-core cpus. Say you launch a 4
>>> task MPI job on a 4-socket dual core Opteron. You'd want to schedule
>>> the tasks on nodes 0, 2, 4, 6 - not 0, 1, 2, 3 to get maximum memory
>>> bandwidth to each MPI task.
>> With the potential for non-trivial logic like this, perhaps the extra
>> work for a real framework would be justified, then.
> I agree.
>>> Also, how would this work with hybrid MPI+threading (either
>>> pthreads
>>> or OpenMP) applications? Let's say you have an 8 or 16 cpu node and
>>> you
>>> start up 2 MPI tasks with 4 compute threads in each task. The
>>> optimum
>>> layout may not be running the MPI tasks on cpu's 0 and 1. Several
>>> hybrid applications that ran on ASC White and now Purple will have
>>> these
>>> requirements.
>> Hum. Good question. The MPI API doesn't really address this -- the
>> MPI API is not aware of additional threads that are created until you
>> call an MPI function (and even then, we're not currently checking
>> which
>> thread is calling -- that would just add latency).
>> What do these applications do right now? Do they set their own
>> processor / memory affinity? This might actually be outside the scope
>> of MPI...? (I'mm not trying to shrug off responsibility, but this
>> might be a case where the MPI simply doesn't have enough information,
>> and to get that information [e.g., via MPI attributes or MPI info
>> arguments] would be more hassle than the user just setting the
>> affinity
>> themselves...?)
> We played around with setting processor affinity in our app a few
> years ago. It got a little ugly, but things have improved since then.
> I was thinking of having the app pass threading info to MPI (via info
> or attributes). This might be outside the scope of MPI now, but this
> should be the responsibility of the parallel programming
> language/method. Making it the apps responsibility to set processor
> affinity seems a bit too much of a low-level worry to put on
> application
> developers.
> Some discussions around what a memory/processor affinity framework
> should look like and be doing is a good starting point.
> - Matt
> _______________________________________________
> devel mailing list
> devel_at_[hidden]