Hi Josh.<div><br></div><div>Thanks for your reply. I&#39;ll tell you what i&#39;m getting now from the executions in the next lines.</div><div>When i run without doing a checkpoint i get this output, and the process don&#39; finish:</div>
<div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div><i>[hmeyer@clus9 whoami]$ /home/hmeyer/desarrollo/ompi-code/binarios/bin/mpirun -np 2 -am ft-enable-cr-recovery ./whoami 10 10</i></div><div><div><i>Antes de MPI_Init</i></div></div><div><div><i>Antes de MPI_Init</i></div>

</div><div><div><i>[clus9:04985] [[41167,0],0] ORTE_ERROR_LOG: Error in file ../../../../../orte/mca/errmgr/hnp/errmgr_hnp_crmig.c at line 287</i></div></div><div><div><i>[clus9:04985] [[41167,0],0] ORTE_ERROR_LOG: Error in file ../../../../../orte/mca/errmgr/hnp/errmgr_hnp_crmig.c at line 287</i></div>

</div><div><div><i>Soy el número 1 (100000000)</i></div></div><div><div><i>Terminando, una instrucción antes del finalize</i></div></div><div><div><i>Soy el número 0 (100000000)</i></div></div><div><div><i>Terminando, una instrucción antes del finalize</i></div>

</div><div><div><i>--------------------------------------------------------------------------</i></div></div><div><div><i>Error: The process below has failed. There is no checkpoint available for</i></div></div><div><div>

<i>       this job, so we are terminating the application since automatic</i></div></div><div><div><i>       recovery cannot occur.</i></div></div><div><div><i>Internal Name: [[41167,1],0]</i></div></div><div><div><i>MCW Rank: 0</i></div>

</div><div><div><i><br></i></div></div><div><div><i>--------------------------------------------------------------------------</i></div></div><div><div><i>[clus9:04985] 1 more process has sent help message help-orte-errmgr-hnp.txt / autor_failed_to_recover_proc</i></div>

</div><div><div><i>[clus9:04985] Set MCA parameter &quot;orte_base_help_aggregate&quot; to 0 to see all help / error messages</i></div></div><div><i><br></i></div></blockquote>If i make a checkpoint in another terminal of the mpirun process, during the execution, i get this output:<div>

<br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><i>[clus9:06105] [[42095,0],0] ORTE_ERROR_LOG: Error in file ../../../../../orte/mca/errmgr/hnp/errmgr_hnp_crmig.c at line 287</i></div>
<div><i>[clus9:06105] [[42095,0],0] ORTE_ERROR_LOG: Error in file ../../../../../orte/mca/errmgr/hnp/errmgr_hnp_crmig.c at line 287</i></div><div><i>[clus9:06107] [[42095,1],1] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file ../../../../orte/mca/grpcomm/base/grpcomm_base_modex.c at line 350</i></div>

<div><i>[clus9:06107] [[42095,1],1] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file ../../../../../orte/mca/grpcomm/bad/grpcomm_bad_module.c at line 323</i></div><div><i>[clus9:06107] pml:ob1: ft_event(Restart): Failed orte_grpcomm.modex() = -26</i></div>

<div><i>[clus9:06106] [[42095,1],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file ../../../../orte/mca/grpcomm/base/grpcomm_base_modex.c at line 350</i></div><div><i>[clus9:06106] [[42095,1],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file ../../../../../orte/mca/grpcomm/bad/grpcomm_bad_module.c at line 323</i></div>

<div><i>[clus9:06106] pml:ob1: ft_event(Restart): Failed orte_grpcomm.modex() = -26</i></div><div><i>--------------------------------------------------------------------------</i></div><div><i>Notice: The job has been successfully recovered from the </i></div>

<div><i>        last checkpoint.</i></div><div><i>--------------------------------------------------------------------------</i></div><div><i>Soy el número 1 (100000000)</i></div><div><i>Terminando, una instrucción antes del finalize</i></div>

<div><i>Soy el número 0 (100000000)</i></div><div><i>Terminando, una instrucción antes del finalize</i></div><div><i>[clus9:06105] 1 more process has sent help message help-orte-errmgr-hnp.txt / autor_recovering_job</i></div>

<div><i>[clus9:06105] Set MCA parameter &quot;orte_base_help_aggregate&quot; to 0 to see all help / error messages</i></div><div><i>[clus9:06105] [[42095,0],0] ORTE_ERROR_LOG: Error in file ../../../../../orte/mca/errmgr/hnp/errmgr_hnp_crmig.c at line 287</i></div>

<div><i>[clus9:06105] [[42095,0],0] ORTE_ERROR_LOG: Error in file ../../../../../orte/mca/errmgr/hnp/errmgr_hnp_crmig.c at line 287</i></div><div><i>[clus9:06107] [[42095,1],1] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file ../../../../orte/mca/grpcomm/base/grpcomm_base_modex.c at line 350</i></div>

<div><i>[clus9:06107] [[42095,1],1] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file ../../../../../orte/mca/grpcomm/bad/grpcomm_bad_module.c at line 323</i></div><div><i>[clus9:06107] pml:ob1: ft_event(Restart): Failed orte_grpcomm.modex() = -26</i></div>

<div><i>[clus9:06106] [[42095,1],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file ../../../../orte/mca/grpcomm/base/grpcomm_base_modex.c at line 350</i></div><div><i>[clus9:06106] [[42095,1],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file ../../../../../orte/mca/grpcomm/bad/grpcomm_bad_module.c at line 323</i></div>

<div><i>[clus9:06106] pml:ob1: ft_event(Restart): Failed orte_grpcomm.modex() = -26</i></div><div><i>[clus9:06105] 1 more process has sent help message help-orte-errmgr-hnp.txt / autor_recovery_complete</i></div><div><i>Soy el número 0 (100000000)</i></div>

<div><i>Terminando, una instrucción antes del finalize</i></div><div><i>Soy el número 1 (100000000)</i></div><div><i>Terminando, una instrucción antes del finalize</i></div><div><i>[clus9:06105] 1 more process has sent help message help-orte-errmgr-hnp.txt / autor_recovering_job</i></div>

<div><i>[clus9:06105] [[42095,0],0] ORTE_ERROR_LOG: Error in file ../../../../../orte/mca/errmgr/hnp/errmgr_hnp_crmig.c at line 287</i></div><div><i>[clus9:06105] [[42095,0],0] ORTE_ERROR_LOG: Error in file ../../../../../orte/mca/errmgr/hnp/errmgr_hnp_crmig.c at line 287</i></div>

<div><i>[clus9:06107] [[42095,1],1] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file ../../../../orte/mca/grpcomm/base/grpcomm_base_modex.c at line 350</i></div><div><i>[clus9:06107] [[42095,1],1] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file ../../../../../orte/mca/grpcomm/bad/grpcomm_bad_module.c at line 323</i></div>

<div><i>[clus9:06106] [[42095,1],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file ../../../../orte/mca/grpcomm/base/grpcomm_base_modex.c at line 350</i></div><div><i>[clus9:06106] [[42095,1],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file ../../../../../orte/mca/grpcomm/bad/grpcomm_bad_module.c at line 323</i></div>

<div><i>[clus9:06106] pml:ob1: ft_event(Restart): Failed orte_grpcomm.modex() = -26</i></div><div><i>[clus9:06107] pml:ob1: ft_event(Restart): Failed orte_grpcomm.modex() = -26</i></div></div></blockquote><div><div><div>
<br>
</div><div>As you can see, it keeps looping on the recover. Then when i try to migrate this processes using ompi-migrate, i get this:</div><div><br></div></div></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;">
<div><div><div><div><i>[hmeyer@clus9 ~]$ /home/hmeyer/desarrollo/ompi-code/binarios/bin/ompi-migrate -x node9 -t node3 18082</i></div></div></div></div><div><div><div><div><i>--------------------------------------------------------------------------</i></div>
</div></div></div><div><div><div><div><i>Error: The Job identified by PID (18082) was not able to migrate processes in this</i></div></div></div></div><div><div><div><div><i>       job. This could be caused by any of the following:</i></div>
</div></div></div><div><div><div><div><i>       - Invalid node or rank specified</i></div></div></div></div><div><div><div><div><i>       - No processes on the indicated node can by migrated</i></div></div></div></div><div>
<div><div><div><i>       - Process migration was not enabled for this job. Make sure to indicate</i></div></div></div></div><div><div><div><div><i>         the proper AMCA file: &quot;-am ft-enable-cr-recovery&quot;.</i></div>
</div></div></div><div><div><div><div><i>--------------------------------------------------------------------------</i></div></div></div></div></blockquote><div><div><div>But, in the terminal where is running the application i get this:</div>
<div><br></div></div></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div><div><div>[hmeyer@clus9 whoami]$ /home/hmeyer/desarrollo/ompi-code/binarios/bin/mpirun -np 2 -am ft-enable-cr-recovery ./whoami 10 10</div>
</div></div></div><div><div><div><div>Antes de MPI_Init</div></div></div></div><div><div><div><div>Antes de MPI_Init</div></div></div></div><div><div><div><div>[clus9:18082] [[62740,0],0] ORTE_ERROR_LOG: Error in file ../../../../../orte/mca/errmgr/hnp/errmgr_hnp_crmig.c at line 287</div>
</div></div></div><div><div><div><div>[clus9:18082] [[62740,0],0] ORTE_ERROR_LOG: Error in file ../../../../../orte/mca/errmgr/hnp/errmgr_hnp_crmig.c at line 287</div></div></div></div><div><div><div><div>--------------------------------------------------------------------------</div>
</div></div></div><div><div><div><div>Warning: Could not find any processes to migrate on the nodes specified.</div></div></div></div><div><div><div><div>         You provided the following:</div></div></div></div><div><div>
<div><div>Nodes: node9</div></div></div></div><div><div><div><div>Procs: (null)</div></div></div></div><div><div><div><div>--------------------------------------------------------------------------</div></div></div></div>
<div><div><div><div>--------------------------------------------------------------------------</div></div></div></div><div><div><div><div>Notice: The processes have been successfully migrated to/from the specified</div></div>
</div></div><div><div><div><div>        machines.</div></div></div></div><div><div><div><div>--------------------------------------------------------------------------</div></div></div></div><div><div><div><div>Soy el número 1 (100000000)</div>
</div></div></div><div><div><div><div>Terminando, una instrucción antes del finalize</div></div></div></div><div><div><div><div>Soy el número 0 (100000000)</div></div></div></div><div><div><div><div>Terminando, una instrucción antes del finalize</div>
</div></div></div><div><div><div><div>--------------------------------------------------------------------------</div></div></div></div><div><div><div><div>Error: The process below has failed. There is no checkpoint available for</div>
</div></div></div><div><div><div><div>       this job, so we are terminating the application since automatic</div></div></div></div><div><div><div><div>       recovery cannot occur.</div></div></div></div><div><div><div><div>
Internal Name: [[62740,1],0]</div></div></div></div><div><div><div><div>MCW Rank: 0</div></div></div></div><div><div><div><div><br></div></div></div></div><div><div><div><div>--------------------------------------------------------------------------</div>
</div></div></div><div><div><div><div>[clus9:18082] 1 more process has sent help message help-orte-errmgr-hnp.txt / autor_failed_to_recover_proc</div></div></div></div><div><div><div><div>[clus9:18082] Set MCA parameter &quot;orte_base_help_aggregate&quot; to 0 to see all help / error messages</div>
</div></div></div><div><br></div></blockquote>I asume that the orte_get_job_data_object is the problem, because it is not obtaining the proper value. <div><br></div><div>If you need more data, just let me know.</div><div>
<br></div><div>Best Regards.</div><div><br></div><div>Hugo Meyer<br><div><div><br></div><div><br><div><div><div><br></div><br><div class="gmail_quote">2011/1/26 Joshua Hursey <span dir="ltr">&lt;<a href="mailto:jjhursey@open-mpi.org" target="_blank">jjhursey@open-mpi.org</a>&gt;</span><br>

<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I found a few more bugs after testing the C/R functionality this morning. I just committed some more C/R fixes in r24306 (things are now working correctly on my test cluster).<br>


  <a href="https://svn.open-mpi.org/trac/ompi/changeset/24306" target="_blank">https://svn.open-mpi.org/trac/ompi/changeset/24306</a><br>
<br>
One thing I just noticed in your original email was that you are specifying the wrong parameter for migration (it is different than the standard C/R functionality for backwards compatibility reasons). You need to use the &#39;ft-enable-cr-recovery&#39; AMCA parameter:<br>


  mpirun -np 2 -am ft-enable-cr-recovery ./whoami 10 10<br>
<br>
If you still get the segmentation fault after upgrading to the current trunk, can you send me a backtrace from the core file? That will help me narrow down on the problem.<br>
<br>
Thanks,<br>
Josh<br>
<div><div></div><div><br>
<br>
On Jan 26, 2011, at 8:40 AM, Hugo Meyer wrote:<br>
<br>
&gt; Josh.<br>
&gt;<br>
&gt; The ompi-checkpoint with his restart now are working great, but the same error persist with ompi-migrate. I&#39;ve also tried using &quot;-r&quot;, but i get the same error.<br>
&gt;<br>
&gt; Best regards.<br>
&gt;<br>
&gt; Hugo Meyer<br>
&gt;<br>
&gt; 2011/1/26 Hugo Meyer &lt;<a href="mailto:meyer.hugo@gmail.com" target="_blank">meyer.hugo@gmail.com</a>&gt;<br>
&gt; Thanks Josh.<br>
&gt;<br>
&gt; I&#39;ve already check te prelink and is set to &quot;no&quot;.<br>
&gt;<br>
&gt; I&#39;m going to try with the trunk head, and then i&#39;ll let you know how it goes.<br>
&gt;<br>
&gt; Best regards.<br>
&gt;<br>
&gt; Hugo Meyer<br>
&gt;<br>
&gt; 2011/1/25 Joshua Hursey &lt;<a href="mailto:jjhursey@open-mpi.org" target="_blank">jjhursey@open-mpi.org</a>&gt;<br>
&gt;<br>
&gt; Can you try with the current trunk head (r24296)?<br>
&gt; I just committed a fix for the C/R functionality in which restarts were getting stuck. This will likely affect the migration functionality, but I have not had an opportunity to test just yet.<br>
&gt;<br>
&gt; Another thing to check is that prelink is turned off on all of your machines.<br>
&gt;  <a href="https://upc-bugs.lbl.gov//blcr/doc/html/FAQ.html#prelink" target="_blank">https://upc-bugs.lbl.gov//blcr/doc/html/FAQ.html#prelink</a><br>
&gt;<br>
&gt; Let me know if the problem persists, and I&#39;ll dig into a bit more.<br>
&gt;<br>
&gt; Thanks,<br>
&gt; Josh<br>
&gt;<br>
&gt; On Jan 24, 2011, at 11:37 AM, Hugo Meyer wrote:<br>
&gt;<br>
&gt; &gt; Hello @ll<br>
&gt; &gt;<br>
&gt; &gt; I&#39;ve got a problem when i try to use the ompi-migrate command.<br>
&gt; &gt;<br>
&gt; &gt; What i&#39;m doing is execute for example the next application in one node of a cluster (both process wil run on the same node):<br>
&gt; &gt;<br>
&gt; &gt; mpirun -np 2 -am ft-enable-cr ./whoami 10 10<br>
&gt; &gt;<br>
&gt; &gt; Then in the same node i try to migrate the processes to another node:<br>
&gt; &gt;<br>
&gt; &gt; ompi-migrate -x node9 -t node3 14914<br>
&gt; &gt;<br>
&gt; &gt; And then i get this message:<br>
&gt; &gt;<br>
&gt; &gt; [clus9:15620] *** Process received signal ***<br>
&gt; &gt; [clus9:15620] Signal: Segmentation fault (11)<br>
&gt; &gt; [clus9:15620] Signal code: Address not mapped (1)<br>
&gt; &gt; [clus9:15620] Failing at address: (nil)<br>
&gt; &gt; [clus9:15620] [ 0] /lib64/libpthread.so.0 [0x2aaaac0b8d40]<br>
&gt; &gt; [clus9:15620] *** End of error message ***<br>
&gt; &gt; Segmentation fault<br>
&gt; &gt;<br>
&gt; &gt; I assume that maybe there is something wrong with the thread level, but i have configured the open-mpi like this:<br>
&gt; &gt;<br>
&gt; &gt; ../configure --prefix=/home/hmeyer/desarrollo/ompi-code/binarios/ --enable-debug --enable-debug-symbols --enable-trace --with-ft=cr --disable-ipv6 --enable-opal-multi-threads --enable-ft-thread --without-hwloc --disable-vt --with-blcr=/soft/blcr-0.8.2/ --with-blcr-libdir=/soft/blcr-0.8.2/lib/<br>


&gt; &gt;<br>
&gt; &gt; The checkpoint and restart works fine, but when i restore an application that has more than one process, this one is restored and executed until the last line before MPI_FINALIZE(), but the processes never finalize, i assume that they never call the MPI_FINALIZE(), but with one process ompi-checkpoint and ompi-restart work great.<br>


&gt; &gt;<br>
&gt; &gt; Best regards.<br>
&gt; &gt;<br>
&gt; &gt; Hugo Meyer<br>
&gt; &gt; _______________________________________________<br>
&gt; &gt; devel mailing list<br>
&gt; &gt; <a href="mailto:devel@open-mpi.org" target="_blank">devel@open-mpi.org</a><br>
&gt; &gt; <a href="http://www.open-mpi.org/mailman/listinfo.cgi/devel" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/devel</a><br>
&gt;<br>
&gt; ------------------------------------<br>
&gt; Joshua Hursey<br>
&gt; Postdoctoral Research Associate<br>
&gt; Oak Ridge National Laboratory<br>
&gt; <a href="http://users.nccs.gov/~jjhursey" target="_blank">http://users.nccs.gov/~jjhursey</a><br>
&gt;<br>
&gt;<br>
&gt; _______________________________________________<br>
&gt; devel mailing list<br>
&gt; <a href="mailto:devel@open-mpi.org" target="_blank">devel@open-mpi.org</a><br>
&gt; <a href="http://www.open-mpi.org/mailman/listinfo.cgi/devel" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/devel</a><br>
&gt;<br>
&gt;<br>
&gt; _______________________________________________<br>
&gt; devel mailing list<br>
&gt; <a href="mailto:devel@open-mpi.org" target="_blank">devel@open-mpi.org</a><br>
&gt; <a href="http://www.open-mpi.org/mailman/listinfo.cgi/devel" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/devel</a><br>
<br>
------------------------------------<br>
Joshua Hursey<br>
Postdoctoral Research Associate<br>
Oak Ridge National Laboratory<br>
<a href="http://users.nccs.gov/~jjhursey" target="_blank">http://users.nccs.gov/~jjhursey</a><br>
<br>
<br>
_______________________________________________<br>
devel mailing list<br>
<a href="mailto:devel@open-mpi.org" target="_blank">devel@open-mpi.org</a><br>
<a href="http://www.open-mpi.org/mailman/listinfo.cgi/devel" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/devel</a><br>
</div></div></blockquote></div><br></div></div>
</div></div></div>