On Jul 15, 2007, at 11:18 PM, Matthew Moskewicz wrote:
>>> i'll probably just continue experimenting on my own for the
>>> moment (tracking
>>> any updates to the main trunk LSF support) to see if i can figure
>>> it out. any
>>> advice the best way to get such back support into trunk, if and
>>> when if exists
>>> / is working?
>> The *best* way would be for you to sign a third-party agreement -
>> see the
>> web site for details and a copy. Barring that, the only option
>> would be to
>> submit the code through either Jeff or I. We greatly prefer the
>> method as it is (a) less burdensome on us and (b) gives you greater
> i'll talk to 'the man' -- it should be okay ... eventually, at
> least ...
See http://www.open-mpi.org/community/contribute/ for details. As an
open project, we always welcome new developers, but we do need to
keep the IP tidy.
>> I can't speak to the motivation behind MPI-2 - the others in the
>> group can
>> do a much better job of that. What I can say is that we started
>> out with a
>> design to support such modes of operation as dynamic farms, but
>> the group
>> has been moving away from it due to a combination of performance
>> reliability, and (frankly) lack of interest from our user
>> community. Our
>> intent now is to cut the RTE back to the basics required to
>> support the MPI
>> standard, including MPI-2 - which arguably says nothing about
>> resource allocation.
> that's true -- dynamic processes can be useful even under a static
> allocation. in fact, in the short term for my particular application,
> i'll probably do just that -- the user picks an initial allocation,
> and then i just do the best i can. hopefully the allocations will be
> 'small enough' to get away without dynamic acquisition for a while (a
FWIW, our experience with the MPI layer has shown that the vast
majority of applications only need a specific set of initial
resources (hosts/cpus) and then just use those. We have seen only a
small class of applications that truly benefit from dynamically
adding / removing resources in the middle of the run. The canonical
manager/worker model fits this criteria (i.e., benefits from
dynamically adding/removing resources), but as you noted, it also
works just fine with a static set of resources. FWIW, I've seen many
MPI applications written with the manager/worker model to ease their
startup with a variable number of nodes (e.g., under a resource
manager) -- they'll just launch as many processes as they get in
their job then then manager/worker from there to "discover" how many
processes they got, use them all, etc.
> beyond that, i guess i'm just one of those guys that thinks
> it's a shame that MPI supplanted pvm so long ago in the first place.
> and yes, i already looked into modifying pvm instead, no thank you ...
A religious argument. ;-) There were certainly good things about
PVM, and MPI managed to take at least some of them.
>> Not to say we won't support it - just indicating that such support
>> will have
>> lower priority and that the system will be designed primarily for
>> priorities. So dynamic resource allocation will have to be
>> considered as an
>> "exception case", with all the attendant implications.
> fair enough. i'm still hoping it won't be too exceptional, really. on
> a related note, perhaps is it possible to 'join' running openMPI jobs
> (using nameservers or whatnot)? if so, then application level
> workarounds are also possible -- and can even be automated if the
> application just launches a whole new copy of itself via whatever
> top-level means was used to launch itself in the first place.
MPI-2 does support the MPI_COMM_JOIN and MPI_COMM_ACCEPT/
MPI_COMM_CONNECT models. We do support this in Open MPI, but the
restrictions (in terms of ORTE) may not be sufficient for you.
Some other random notes in no particular order:
- As you noted, the LSF support is *very* new; it was just added last
- It also likely doesn't work yet; we started the integration work
and ran into a technical issue that required further discussion with
Platform. They're currently looking into it; we stopped the LSF work
in ORTE until they get back to us.
- FWIW, one of the main reasons OMPI/ORTE didn't add extensive/
flexible support for dynamic addition of resources was the potential
for queue time. Many systems run "full" all the time, so if you try
to acquire more resources, you could just sit in a queue for minutes/
hours/days/weeks before getting nodes. While it is certainly
possible to program with this model, we didn't really want to get
into the rats nest of corner cases that this would entail, especially
since very few users are asking for it.
- That being said, MPI_THREAD_MULTIPLE and MPI_COMM_SPAWN *might*
offer a way out here. But I think a) THREAD_MULTIPLE isn't working
yet (other OMPI members are working on this), and b) even when
THREAD_MULTIPLE works, there will be ORTE issues to deal with
(canceling pending resource allocations, etc.). Ralph mentioned that
someone else is working on such things on the TM/PBS/Torque side; I
haven't followed that effort closely.
> well, certainly part of the issue is the need (or at least strong
> preference) to support 6.2 -- but read on.
> hmm, i'll need to review the APIs in more detail, but here is my
> current understanding:
> there appear to be some overlaps between the ls_* and lsb_* functions,
> but they seem basically compatible as far as i can tell. almost all
> the functions have a command line version as well, for example:
> lsb_getalloc()/none and lsb_launch()/blaunch are new with LSF 7.0, but
> appear to just be a different (simpler) interface to existing
> functionality in the LSB_* env vars and the ls_rexec()/lsgrun commands
> -- although, as you say, perhaps platform will hook or enhance them
> later. but, the key issue is that lsb_launch() just starts tasks -- it
> does not perform or interact with the queue or job control (much?).
> so, you can't use these functions to get an allocation in the first
> place, and you have to be careful not to use them as a way around the
> queuing system.
> [ as a side note, the function ls_rexecv()/lsgrun is the one i have
> heard admins do not like because it can break queuing/accounting, and
> might try to disable somehow. i don't really buy that, because it's
> not you can disable it and have the system still work, since (as
> above) || job launching depends on it. i guess if you really don't
> care about || launching maybe you could. but, if used properly after a
> proper allocation i don't think there should (or even can) be a
> problem. ]
> so, lsb_submit()/bsub is a combination allocate/launch -- you specify
> the allocation size you want, and when it's all ready, it runs the
> 'job' (really the job launcher) only on one (randomly chosen) 'head'
> node from the allocation, with the env vars set so the launcher can
> use ls_rexec/lsgrun functions to start the rest of the job. there are
> of course various script wrappers you can use (mpijob, pvmjob, etc)
> instead of your 'real job'. then, i think lsf *should* try to track
> what processes get started via the wrapper / head process so it knows
> they are part of the same job. i dunno if it really does that -- but,
> my guess is that at the least it assumes the allocation is in use
> until the original process ends. in any case, the wrapper / head
> process examines the environment vars and uses ls_rexec()/lsgrun or
> the like to actually run N copies of the 'real job' executable. in
> 7.0, it can conveniently use lsb_getalloc() and lsb_launch(), but that
> doesn't really change any semantics as far as i know. one could
> imaging that calling lsb_launch() instead of ls_rexec() might be
> preferable from a process tracking point of view, but i don't see why
> Platform couldn't hook ls_rexec() just as well as lsb_launch().
> i really need to get a little more confidence on that issue, since
> it's what determines what actions will (or perhaps already do in
> practice) 'break' the queuing/reporting system.
> there are some 'allocate only' functions as well, such as
> ls_placereq()/lsplace -- these can just return a host list / set the
> env vars without running anything at first. apparently, you need to
> run something 'soon' on the resultant hosts or the load balancer might
> get confused and reuse them. also, since this doesn't seem to go
> through the queues, it's probably not a viable set of functions to
> really use. a red herring, as far as i'm concerned.
> there is also an lsb_runjob() that is similar to lsb_launch(), but for
> an already submitted job. so, if one were to lsb_sumbit() with an
> option set to never launch it automatically, and then one were to run
> lsb_runjob(), you can avoid the queue and/or force the use of certain
> hosts? i guess this is also not a good function to use, but at least
> the queuing system would be aware of any bad behavior (queue skipping
> via ls_placereq() to get extra hosts, for instance) in this case ...
> there does *not* appear to be an option to lsb_submit() that allows a
> non-blocking programmatic callback when allocation is complete. if
> there was, it would need to deal with process tracking issues, or
> maybe just merge the old and new jobs somehow in that case.
> so to speak to the original point, it would indeed be nice to be able
> to do additional allocations (and then an lsb_launch) with a simple
> programmatic interface for completeness, but i don't see one. however,
> lsb_submit() is pretty close -- it makes a 'new' job, but i think
> that's okay. the initial daemon that gets run on the 'head' (i.e.
> randomly chosen) node of the new job will run an lsb_launch() or
> similar to start up the remaining N-1 daemons as children -- thus
> hopefully keeping the queuing system and process tracking happy. or
> you could use some LSF option / wrapper script to tell it to run the
> same daemon on all N hosts for you, if a some suitable option/wrapper
> exists anyway. so, in summary lsb_sumit() does allocation + one
> (non-optional) launch on allocation completion. lsb_launch() (or
> similar) does only launching, should probably only be run from the
> single process started from an lsb_submit(), and should only launch
> things on the allocation given by lsb_getalloc() (or env vars).
I am certainly not an expert on LSF (nor its API) -- I only started
using it last week! Do you have any contacts to ask at Platform?
They would likely be the best ones to discuss this with.