Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Open MPI task scheduler
From: Jack Bryan (dtustudy68_at_[hidden])
Date: 2010-06-20 14:44:24

Hi, Matthieu:
Thanks for your help.
Most of your ideas show that what I want to do.
My scheduler should be able to be called from any C++ program, which can put a list of tasks to the scheduler and then the scheduler distributes the tasks to other client nodes.
It may work like in this way:
while(still tasks available) { myScheduler.push(tasks); myScheduler.get(tasks results from client nodes);}

My cluster has 400 nodes with Open MPI. The tasks should be transferred b y MPI protocol.
I am not familiar with RPC Protocol.
If I use Boost.ASIO and some Python/GCCXML script to generate the code, it can be called from C++ program on Open MPI cluster ?
I cannot find the skeletton on your blog.
Would you please tell me where to find it ?
I really appreciate your help.

June 20 2010
> Date: Sun, 20 Jun 2010 20:13:14 +0200
> From: matthieu.brucher_at_[hidden]
> To: users_at_[hidden]
> Subject: Re: [OMPI users] Open MPI task scheduler
> Hi Jack,
> What you are seeking is the client/server pattern. Have one node act
> as a server. It will create a list of tasks or even a graph of tasks
> if you have dependencies, and then create clients that will connect to
> the server with an RPC protocol (I've done this with a SOAP+TCP
> protocol, the severance of the TCP connection meaning that the client
> is dead and that its task should be recycled, ités easy to do with
> Boost.ASIO and some Python/GCCXML scripts to automatically generate
> your code, I've written a skeletton on my blog). You may even have
> clients with different sizes or capabilities and tell the server what
> each client can do, and then the server may dispatch appropriate
> tickets to the clients.
> Each client and server can be a MPI process, you don't have to create
> all clients inside one MPI process (you may use several if the
> smallest resource your batch scheduler allocates is bigger that one of
> your tasks). With a batch scheduler, it's better to allocate your
> tasks as small as possible so that you can balance the resources you
> need.
> Matthieu
> 2010/6/20 Jack Bryan <dtustudy68_at_[hidden]>:
> > Hi, all:
> > I need to design a task scheduler (not PBS job scheduler) on Open MPI
> > cluster.
> > I need to parallelize an algorithm so that a big problem is decomposed into
> > small tasks, which can be distributed
> > to other worker nodes by the Scheduler and after being solved, the results
> > of these tasks are returned to the manager node with the Scheduler, which
> > will distribute more tasks on the base of the collected results.
> > I need to use C++ to design the scheduler.
> > I have searched online and I cannot find any scheduler available
> > for this purpose.
> > Any help is appreciated.
> > thanks
> > Jack
> > June 19 2010
> > ________________________________
> > Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox.
> > Learn more.
> > _______________________________________________
> > users mailing list
> > users_at_[hidden]
> >
> >
> --
> Information System Engineer, Ph.D.
> Blog:
> LinkedIn:
> _______________________________________________
> users mailing list
> users_at_[hidden]
The New Busy think 9 to 5 is a cute idea. Combine multiple calendars with Hotmail.