Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Open MPI task scheduler
From: Matthieu Brucher (matthieu.brucher_at_[hidden])
Date: 2010-06-21 03:14:18


2010/6/21 Jack Bryan <dtustudy68_at_[hidden]>:
> Hi,
> thank you very much for your help.
> What is the meaning of " must find a system so that every task can be
> serialized in the same form." What is the meaning of "serize " ?

Serialize is the process of converting an object instance into a
text/binary stream, and to create a new object instance from this
stream. It allows communication of data between processes. With MPI,
you send one data type after another, with serialization, you send
everything in one big chunk.

> I have no experience of programming with python and XML.

Python is not mandatory at all. I use it to automate the wrappers/SOAP
files generation, and to talk to the daemon. You can do this is any
language you are comfortable with.

> I have studied your blog.
> Where can I find a simple example to use the techniques you have said ?

If you are looking for RPC samples, you can ask google with just SOAP
as key, it will find several tutorials on how this works. As Jody
said, you may want something simplier if you can have all tasks in one
MPI process, but once you go on a big cluster with variable resources,
you will be stuck.

> For exmple, I have 5 task (print "hello world !").
> I want to use 6 processors to do it in parallel.
> One processr is the manager node who distributes tasks and other 5
> processors
> do the printing jobs and when they are done, they tell this to the manager
> noitde.

In this case, you have your daemon working in parallel from the batch
scheduler, and then each process asks the daemon for a new ticket. You
may add tickets by talking to the dameon directly, without having to
launch a new job.

> Boost.Asio is a cross-platform C++ library for network and low-level I/O
> programming. I have no experiences of using it. Will it take a long time to
> learn
> how to use it ?

The longest time will not be to master Boost, but more to understand
how to create your TCP server and to serialize your parameters.

> If the messages are transferred by SOAP+TCP, how the manager node calls it
> and push task into it ?

You have to think of SOAP + TCP as just a simple function call that
hides everything. From the client node point of view, it's a simple
function call server.get_ticket(). The manager node will be talked to
by different kind of programs: task programs or by a program that will
push tickets. The latter one will just be another function call like
this in C++:

std::vector<std::string> tickets;
daemon.connect(somewhere, port);
daemon.add_tickets(tickets);

> Do I need to install SOAP+TCP on my cluster so that I can use it ?

As Jody said, you can do things with MPI directly. I would not
recommand it, but this will help you with a fast solution. You will
have to use some MPI2 calls to create a socket on the daemon to talk
to it, and in fact, you will have to create exactly what I proposed,
but less portable.

Matthieu

-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher