Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] RFC: hostfile setting of #slots
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2012-09-04 10:30:30


How about adding (yet another) MCA param: orte_<whatever>_default_hostfile_slots, and it can take multiple different values:

- an integer >0
- num_cores
- num_hwthreads
- num_sockets
- ...could have others here, but I don't know if it's actually useful to be overly flexible here

Today's current default behavior would be to set this value to "1". But it could easily change to "num_cores" (for example).

On Sep 2, 2012, at 11:33 AM, Ralph Castain wrote:

> Guess I should have emphasized the "IF" more - I'm talking about setting the slots ONLY in the case where the user didn't provide that information. If the user provides it, then that is what we use. In OMPI, we *always* accept the user as the ultimate decider.
>
> As things stand, we assume slots=1. This is just as arbitrary as you can get.
>
> My point was that now that we use hwloc to discover the number of cpus in the system, we could do something more intelligent than just set it to 1 *in the case where we are given no info*.
>
> Note that there is no intention to set binding policy here. All this impacts is how many procs are mapped to a given node when we use the byslot mapping algo.
>
> Does that help?
>
> On Sep 2, 2012, at 6:20 AM, Kenneth A. Lloyd <kenneth.lloyd_at_[hidden]> wrote:
>
>> I should note that we only virtualize the private cloud / management nodes
>> over our HPC. The HPC is not virtualized as such.
>>
>> Ken
>>
>> -----Original Message-----
>> From: devel-bounces_at_[hidden] [mailto:devel-bounces_at_[hidden]] On
>> Behalf Of Kenneth A. Lloyd
>> Sent: Sunday, September 02, 2012 7:14 AM
>> To: 'Open MPI Developers'
>> Subject: Re: [OMPI devel] RFC: hostfile setting of #slots
>>
>> This is a tricky issue, isn't it? With the differences between AMD & Intel,
>> and between base operating systems "touching" & heaps (betw. Linux &
>> Windows), and various virtual machines schemes- we have opted for an
>> "outside the main code path" solution to get deterministic results. But that
>> is as things are now. Who knows how stuff like AVX2 / memory mapping - or
>> the next new thing - will affect this? This is similar to issues we've
>> found with CPU/GPU memory & affinity mapping over IB. The basis of the
>> decision is (as is often) how much do you trust the user to do the right
>> thing? What happens if you are wrong?
>>
>> Only my opinion.
>>
>> Ken
>>
>> -----Original Message-----
>> From: devel-bounces_at_[hidden] [mailto:devel-bounces_at_[hidden]] On
>> Behalf Of Ralph Castain
>> Sent: Saturday, September 01, 2012 3:54 AM
>> To: Open MPI Developers
>> Subject: [OMPI devel] RFC: hostfile setting of #slots
>>
>> This is not a notice of intended change, but rather a solicitation for
>> comment.
>>
>> We currently default the number of slots on a host provided via hostfile or
>> -host to 1. This is a historical "feature" driven by the fact that our
>> initial launch system spawned daemons on the remote nodes after we had
>> already mapped the processes to them - so we had no way of guessing a
>> reasonable value for the number of slots on any node.
>>
>> However, the "vm" launch method starts daemons on every node prior to doing
>> the mapping, precisely so we can use the topology in the mapping algorithm.
>> This creates the possibility of setting the number of slots on a node to the
>> number of cpus (either cores or hardware threads, depending on how that flag
>> is set) IF it wasn't provided in the hostfile.
>>
>> This would have an impact on the default "byslot" mapping algorithm. With
>> only one slot/node, byslot essentially equates to bynode mapping. So a
>> user-provided hostfile that doesn't give any info on the number of slots on
>> a node effectively changes the default mapping algorithm to "bynode". This
>> change would alter that behavior and retain a "byslot" algorithm, with the
>> number of slots being the number of cpus.
>>
>> I have a use-case that would benefit from making the change, but can handle
>> it outside of the main code path. However, if others would also find it of
>> use, I can add it to the main code path, either as the default or via MCA
>> param.
>>
>> Any thoughts?
>> Ralph
>>
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/