Hi, Matthieu:

Thanks for your help. 

Most of your ideas show that what I want to do. 

My scheduler should be able to be called from any C++ program, which can put 
a list of tasks to the scheduler and then the scheduler distributes the tasks to other client nodes. 

It may work like in this way: 

while(still tasks available) {
myScheduler.get(tasks results from client nodes);

My cluster has 400 nodes with Open MPI. The tasks should be transferred b y MPI protocol. 

I am not familiar with  RPC Protocol.  

If I use Boost.ASIO and some Python/GCCXML script to generate the code, it can be 
called from C++ program on Open MPI cluster ? 

I cannot find the skeletton on your blog. 

Would you please tell me where to find it ?

I really appreciate your help. 


June 20 2010

> Date: Sun, 20 Jun 2010 20:13:14 +0200
> From: matthieu.brucher@gmail.com
> To: users@open-mpi.org
> Subject: Re: [OMPI users] Open MPI task scheduler
> Hi Jack,
> What you are seeking is the client/server pattern. Have one node act
> as a server. It will create a list of tasks or even a graph of tasks
> if you have dependencies, and then create clients that will connect to
> the server with an RPC protocol (I've done this with a SOAP+TCP
> protocol, the severance of the TCP connection meaning that the client
> is dead and that its task should be recycled, itÚs easy to do with
> Boost.ASIO and some Python/GCCXML scripts to automatically generate
> your code, I've written a skeletton on my blog). You may even have
> clients with different sizes or capabilities and tell the server what
> each client can do, and then the server may dispatch appropriate
> tickets to the clients.
> Each client and server can be a MPI process, you don't have to create
> all clients inside one MPI process (you may use several if the
> smallest resource your batch scheduler allocates is bigger that one of
> your tasks). With a batch scheduler, it's better to allocate your
> tasks as small as possible so that you can balance the resources you
> need.
> Matthieu
> 2010/6/20 Jack Bryan <dtustudy68@hotmail.com>:
> > Hi, all:
> > I need to design a task scheduler (not PBS job scheduler) on Open MPI
> > cluster.
> > I need to parallelize an algorithm so that a big problem is decomposed into
> > small tasks, which can be distributed
> > to other worker nodes by the Scheduler and after being solved, the results
> > of these tasks are returned to the manager node with the Scheduler, which
> > will distribute more tasks on the base of the collected results.
> > I need to use C++ to design the scheduler.
> > I have searched online and I cannot find any scheduler available
> > for this purpose.
> > Any help is appreciated.
> > thanks
> > Jack
> > June 19  2010
> > ________________________________
> > Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox.
> > Learn more.
> > _______________________________________________
> > users mailing list
> > users@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
> --
> Information System Engineer, Ph.D.
> Blog: http://matt.eifelle.com
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> _______________________________________________
> users mailing list
> users@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

The New Busy think 9 to 5 is a cute idea. Combine multiple calendars with Hotmail. Get busy.