On Aug 30, 2011, at 9:26 AM, John Hearns wrote:
> On 30 August 2011 02:55, Ralph Castain <rhc_at_[hidden]> wrote:
>> Instead, all used dynamic requests - i.e., the job that was doing a comm_spawn would request resources at the time of the comm_spawn call. I would pass the request to Torque, and if resources were available, immediately process them into OMPI and spawn the new job. If resources weren't available, I simply returned an error to the program so it could either (a) terminate, or (b) wait awhile and try again. One of the groups got ambitious and supported non-blocking requests (generated a callback to me with resources when they became available). Worked pretty well - might work even better once we get non-blocking MPI_Comm_spawn.
>> I believe they generally were happy with the results, though I think some of them wound up having Torque "hold" a global pool of resources to satisfy such requests, just to avoid blocking progress on the job while waiting for comm_spawn resources.
> Quite often on a larger cluster there are several jobs running
> simultaneously - and you configure the batch scheduler to select
> groups of nodes which are physically close to each other as you get a
> bit more performance that way.
> However, if (say) a node is down for maintenance it can knock these
> patterns out. Could we forsee a dynamic movement of MPI jobs which
> move back to a node when it is replaced?
The run-time in OMPI already supports that - just waiting for the MPI layer to handle that situation.
> users mailing list