Dear Jeff<br><br>Thanks for your help. <br>Unfortunately, after I thoroughly examined entire cluster, I found a bad node with busted hard drive. That&#39;s the reason why this job hanged. <br>Also, when this job is sent with one bad node among the machinefile, neither the openmpi nor my program gives me any error messages. That&#39;s why I can&#39;t find the reason for job hanged.<br>
<br>Best regard<br><br><div class="gmail_quote">2009/4/22 Jeff Squyres <span dir="ltr">&lt;<a href="mailto:jsquyres@cisco.com">jsquyres@cisco.com</a>&gt;</span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">On Apr 21, 2009, at 11:01 AM, Tsung Han Shie wrote:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I tried to increase speed of a program with openmpi-1.1.3<br>
</blockquote>
<br></div>
Did you mean 1.1.3 or 1.3.1?<div class="im">I mean 1.1.3.<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
by adding following 4 parameters into openmpi-mca-params.conf file.<br>
<br>
mpi_leave_pinned=1<br>
btl_openib_eager_rdma_num=128<br>
btl_openib_max_eager_rdma=128<br>
btl_openib_eager_limit=1024<br>
</blockquote>
<br></div>
If you meant 1.3.1 above, please see the following message about an important bug in 1.3 and 1.3.1 with the use of mpi_leave_pinned:<br>
<br>
    <a href="http://www.open-mpi.org/community/lists/announce/2009/03/0029.php" target="_blank">http://www.open-mpi.org/community/lists/announce/2009/03/0029.php</a><div class="im"><br>
<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
and then, I ran my program twice(124 processes on 31 nodes). one with &quot;mpi_leave_pinned=1&quot;, another with &quot;mpi_leave_pinned=0&quot;.<br>
All of them were stopped abnormally with &quot;ctrl+c&quot; and &quot;killall -9 &lt;program&gt;&quot;.<br>
</blockquote>
<br></div>
Why -- did they hang?</blockquote><div>    I just fun my program for a few steps to see the speed and then I killed it. <br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
After that, I couldn&#39;t start to run that program again.<br>
</blockquote>
<br></div>
What exactly was the error?</blockquote><div>    There are not any messages.<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="im">
<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I checked every nodes with &quot;free -m&quot; and I found that huge amount of cached memory were used in each nodes.<br>
Could this situation be caused by those 4 parameters? IS there anyway to free theme?<br>
</blockquote>
<br>
<br></div>
Probably not.<br>
<br>
Can you send all the information listed here:<br>
<br>
    <a href="http://www.open-mpi.org/community/help/" target="_blank">http://www.open-mpi.org/community/help/</a><br><font color="#888888">
<br>
-- <br>
Jeff Squyres<br>
Cisco Systems<br>
<br>
_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@open-mpi.org" target="_blank">users@open-mpi.org</a><br>
<a href="http://www.open-mpi.org/mailman/listinfo.cgi/users" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/users</a><br>
</font></blockquote></div><br>