Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Change in communication between process (RMAPS)
From: Joshua Hursey (jjhursey_at_[hidden])
Date: 2011-01-06 15:46:18


So I can point you to some of the work that I did while at Indiana University to support process migration in Open MPI in a coordinated manner. This should introduce you to some of the internal pieces that fit together to provide this support.

The transparent C/R in Open MPI webpage from IU is a good place to start:
   http://osl.iu.edu/research/ft/ompi-cr/index.php

>From there you will find a link to a couple papers that should get you started. In particular "A Composable Runtime Recovery Policy Framework Supporting Resilient HPC Applications" discusses how the ORTE ErrMgr framework was used (initially) to provide process migration and automatic recovery. The actual code in the Open MPI trunk is slightly different. Instead of using different components of the ErrMgr framework (i.e., autor, crmig, stable) we just rolled it all into the existing components (i.e., hnp, orted, app). But all the code can be found in those component directories.

If you want a more general overview of the C/R system in Open MPI, I would start with the paper "The Design and Implementation of Checkpoint/Restart Process Fault Tolerance for Open MPI" which provides a high level view of the architecture (combined with the paper above you will have a fairly complete picture of the design). The C/R infrastructure currently only supports coordinated C/R, but was designed to be more extensible. So if you are looking into uncoordinated C/R techniques you may find that many of the C/R frameworks in Open MPI can be reused.

That should get you started. Let us know if you have any further questions.

-- Josh

On Jan 6, 2011, at 3:19 PM, Hugo Meyer wrote:

> Thanks for the reply and don't worry about the delay.
>
> Yeah, i supposse it wouln't be easy :(.
> But my final goal is what you are mentioning, is to stop one particular process (previously checkpointed) and the migrate it to another place (node, core, slot, etc.) and restart it there, but without making a coordinated checkpoint. I just need to checkpoint processes in an uncoordinated way, and move them.
>
> Where can i see something about process migration in the code? or something that could guide me.
>
> Greetings.
>
> Hugo Meyer
>
> 2011/1/6 Jeff Squyres <jsquyres_at_[hidden]>
> Sorry for the delay; you wrote while many of us were on vacation and we're just now starting to catch up on past mails...
>
> I'm not entirely sure what you're trying to do. It sounds like you're trying to replace one process with another. That's quite complicated; there will be a lot of changes required in the code base to do this.
>
> - you'll need to notify the ORTE subsystem of the process change
> - this notification will likely need to span multiple processes
> - all MPI processes will need to quiesce their communications, disconnect, and reconnect
> - ...and probably other things
>
> That being said, you might be able to leverage some of the work that's been done with checkpoint/restart/migration. It's not entirely the same thing that you're doing, but it's at least similar (quiesce networks, [pretend to] move a process from location A to location B, etc.).
>
>
>
> On Dec 28, 2010, at 7:03 AM, Hugo Meyer wrote:
>
> > Hello to all.
> >
> > I'm new in the forum, at least is the first time i write.
> >
> > I'm working with open mpi and I would do a little experiment, i will try to pass one process by another process.
> >
> > For example, assuming that there are 2 processes that are communicating say rank 1 and 2. And there is a process of rank 3, I would like the rank 3 (it could be assumed that this node is marked down at the initial hostfile) took the place of rank 2, and rank 1 still think that he is communicating with rank 2 when in fact is communicating with the rank 3.
> >
> > I guess I'll have to modify tables as orte_job_map_t and orte_proc_t, but I wanted to know if someone already has experience doing something similar, and can guide me at least.
> >
> > The communication between processes, in principle, would be irrelevant, so i will not need to use checkpoints / restarts for now.
> >
> > Greetings
> >
> > Hugo Meyer
> > _______________________________________________
> > devel mailing list
> > devel_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
> --
> Jeff Squyres
> jsquyres_at_[hidden]
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

------------------------------------
Joshua Hursey
Postdoctoral Research Associate
Oak Ridge National Laboratory
http://users.nccs.gov/~jjhursey