In a word, no. If a node crashes, OMPI will abort the currently-running job if it had processes on that node. There is no current ability to &quot;ride-thru&quot; such an event.<div><br></div><div>That said, there is work being done to support &quot;ride-thru&quot;. Most of that is in the current developer&#39;s code trunk, and more is coming, but I wouldn&#39;t consider it production-quality just yet.</div>
<div><br></div><div>Specifically, the code that does what you specify below is done and works. It is recovery of the MPI job itself (collectives, lost messages, etc.) that remains to be completed.</div><div><br><br><div class="gmail_quote">
On Thu, Sep 23, 2010 at 7:22 AM, Andrei Fokau <span dir="ltr">&lt;<a href="mailto:andrei.fokau@neutron.kth.se">andrei.fokau@neutron.kth.se</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="gmail_quote"><font face="arial, helvetica, sans-serif"><span style="font-size:12.5px;border-collapse:collapse"><div><font face="arial,helvetica,sans-serif">Dear users,</font></div>
<div><font face="arial,helvetica,sans-serif"><br></font></div><div><font face="arial,helvetica,sans-serif">Our cluster has a number of nodes which have high probability to crash, so it happens quite often that calculations stop due to one node getting down. May be you know if it is possible to block the crashed nodes during run-time when running with OpenMPI? I am asking about principal possibility to program such behavior. Does OpenMPI allow such dynamic checking? The scheme I am curious about is the following:</font></div>



<div><font face="arial,helvetica,sans-serif"><br></font></div><div><font face="arial,helvetica,sans-serif">1. A code starts its tasks via mpirun on several nodes</font></div><div><font face="arial,helvetica,sans-serif">2. At some moment one node gets down</font></div>



<div><font face="arial,helvetica,sans-serif">3. The code realizes that the node is down (the results are lost) and excludes it from the list of nodes to run its tasks on</font></div><div><font face="arial,helvetica,sans-serif">4. At later moment the user restarts the crashed node</font></div>



<div><font face="arial,helvetica,sans-serif">5. The code notices that the node is up again, and puts it back to the list of active nodes</font></div><div><br></div><div><font face="arial,helvetica,sans-serif"><br></font></div>



<div><font face="arial,helvetica,sans-serif">Regards,</font></div><div><font face="arial,helvetica,sans-serif">Andrei</font></div></span></font>
</div><br>
<br>_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@open-mpi.org">users@open-mpi.org</a><br>
<a href="http://www.open-mpi.org/mailman/listinfo.cgi/users" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/users</a><br></blockquote></div><br></div>