Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] RFC: hostfile setting of #slots
From: Kenneth A. Lloyd (kenneth.lloyd_at_[hidden])
Date: 2012-09-02 09:13:55

This is a tricky issue, isn't it? With the differences between AMD & Intel,
and between base operating systems "touching" & heaps (betw. Linux &
Windows), and various virtual machines schemes- we have opted for an
"outside the main code path" solution to get deterministic results. But that
is as things are now. Who knows how stuff like AVX2 / memory mapping - or
the next new thing - will affect this? This is similar to issues we've
found with CPU/GPU memory & affinity mapping over IB. The basis of the
decision is (as is often) how much do you trust the user to do the right
thing? What happens if you are wrong?

Only my opinion.


-----Original Message-----
From: devel-bounces_at_[hidden] [mailto:devel-bounces_at_[hidden]] On
Behalf Of Ralph Castain
Sent: Saturday, September 01, 2012 3:54 AM
To: Open MPI Developers
Subject: [OMPI devel] RFC: hostfile setting of #slots

This is not a notice of intended change, but rather a solicitation for

We currently default the number of slots on a host provided via hostfile or
-host to 1. This is a historical "feature" driven by the fact that our
initial launch system spawned daemons on the remote nodes after we had
already mapped the processes to them - so we had no way of guessing a
reasonable value for the number of slots on any node.

However, the "vm" launch method starts daemons on every node prior to doing
the mapping, precisely so we can use the topology in the mapping algorithm.
This creates the possibility of setting the number of slots on a node to the
number of cpus (either cores or hardware threads, depending on how that flag
is set) IF it wasn't provided in the hostfile.

This would have an impact on the default "byslot" mapping algorithm. With
only one slot/node, byslot essentially equates to bynode mapping. So a
user-provided hostfile that doesn't give any info on the number of slots on
a node effectively changes the default mapping algorithm to "bynode". This
change would alter that behavior and retain a "byslot" algorithm, with the
number of slots being the number of cpus.

I have a use-case that would benefit from making the change, but can handle
it outside of the main code path. However, if others would also find it of
use, I can add it to the main code path, either as the default or via MCA

Any thoughts?

devel mailing list