If you are interested in the ongoing scalability work, or in the RML/OOB in
ORTE, please read on - otherwise, feel free to hit "delete".
As many of you know, we have been working towards solving several problems
that affect our ability to operate at large scale. Some of the required
modifications to the code base have recently been applied to the trunk.
We have known since it was originally written over two years ago that the
OOB contained some inherent scalability limits. For example, the system
immediately upon opening obtains contact info for all daemons in the
universe, opens sockets to them, and sends an initial message to them. It
then does the same with all the application processes in its job.
As a result, for a 2000 process job running on 500 nodes, each application
process will immediately open and communicate across 2501 sockets (2000
procs + 500 daemons [one per node] + the HNP) during the startup phase.
If you really want to imagine some fun, now have that job comm_spawn 500
processes across the 500 nodes, and *don't* reuse daemons. As each new
daemon is spawned, every process in the original job (including the original
daemons) is notified, loads the new contact info for that daemon, opens a
socket to it, and does an "ack" comm. After all 500 new daemons are running,
they now launch the 500 new procs, each of which gets the info on 1000
daemons plus the info for 2000 parents and 500 peers, and immediately opens
1000 daemons + 2000 parents + 500 peers + 1 HNP = 3501 sockets!
This was acceptable for small jobs, but causes considerable delay during
startup for large jobs. A few other OOB operational characteristics further
exacerbate the problem - I will detail those in a document on the wiki to
help foster greater understanding.
Jeff Squyres and I are about to begin a major revision of the RML/OOB code
to resolve these problems. We will be using a staged approach to the effort:
1. separate the OOB's actions for loading contact info from actually opening
a socket to a process. Currently, the OOB immediately opens a socket and
performs an "ack" communication whenever contact info for another process is
loaded into it. In addition, the OOB immediately subscribes to the job
segment of the provided process, requesting that this process be alerted to
*any* change in OOB contact info to any process in that job. These actions
need to be separated out.
2. revise the RML/OOB init/open procedure. These are currently interwoven in
a manner that causes the OOB to execute registry operations that are not
needed (and actually cause headaches) during orte_init. The procedure will
be revised so that connections to the HNP and to the process' local orted
are opened, but all other contact info (e.g., for the other procs in the
job) is simply loaded into the OOB's contact tables, but no sockets opened
until first communication.
3. revise the xcast procedure so that it relays via the daemons and not the
application processes. For systems that do not use our daemons, alternative
mechanisms will be developed.
At some point in the future, a fully routable OOB will be developed to
remove the need for so many sockets on each application process. For now,
these steps should improve our startup time considerably.
With some luck and (hopefully) not too many conflicting priorities, Jeff and
I may complete this work by Christmas - more likely, though, is sometime
early in Jan. We will be working on a tmp branch, but you may see some
transfer of code to the trunk as we progress.
As always, feel free to comment and/or make suggestions!