This may seem like an odd query (or not; perhaps it has been brought up
before). My work recently involves HPC usability i.e. making things
easier for new users by abstracting away the scheduler. I've been
working with DRMAA for interfacing with DRMs and it occurred to me: what
would be the advantage to letting the scheduler itself handle farming
out MPI processes as individual tasks rather than having a wrapper like
mpirun to handle this task via ssh/rsh/etc.?
I thought about MPI2's ability to do dynamic process management and how
scheduling environments tend to allocate static pools of resources for
parallel tasks. A DRMAA-driven MPI would be able to request that the
scheduler launch these tasks as resources become available enabling
scheduled MPI jobs to dynamically add and remove processors during
execution. Several applications that I have worked with come to mind,
where pre-processing and other tasks are non-parallel whereas the
various solvers are. Being able to dynamically spawn processes based on
where you are in this work-flow could be very useful here.
It also occurred to me that commercial application vendors tend to
roll-their-own when it comes to integrating their applications with an
MPI library. I've seen applications use HP-MPI, MPICH, MPICH2,
Intel-MPI, (and thankfully, recently) OpenMPI and then proceed to
butcher the execution mechanisms to such an extent that it makes
integration with common DRM systems quite a task. With the exception of
OpenMPI, none of these libraries provides turn-key compatibility with
most of the major DRMs and each require some degree of manual
integration and testing for use in a multi-user production environment.
I would think that vendors would be falling over themselves to integrate
OpenMPI with their applications for this very reason alone. Instead,
some opt to develop their own scheduling environments! Don't they have
bean counters that sit around and gripe about duplicated work?
Then it occurred to me: with the exception of being able to easily
launch an MPI job with OpenMPI, the ability to monitor it from within
the application is still dependent on the vendor integrating with
various DRMs! This is another area where a DRMAA RAS can come in handy.
There are nice bindings for monitoring tasks and getting an idea of
where you are in execution without having to resort to kludgey
shell-script wrappers tailing output files.
Anyway, its been a frustrating couple of weeks dealing with several
commercial vendors and integrating their applications with our DRM and
my mind has been trying to think of a solution that could save all of us
a lot of work (though, at the same time, raise job security concerns in
such turbulent times ;-/ ). What say you, MPI experts?
Many thanks for your thoughts!