Dear List,<div><br></div><div>I have a hybrid MPI/Pthreads program named &quot;my_hybrid_app&quot;, this program is memory-intensive and take advantage of multi-threading to improve memory throughput. I run &quot;my_hybrid_app&quot; on two machines, which have same hardware configuration but different OS and GCC. The problem is: when I run &quot;my_hybrid_app&quot; with one process, two machines behaves the same: the more number of threads, the better the performance; however, when I run &quot;my_hybrid_app&quot; with two or more processes. The first machine still increase performance with more threads, the second machine degrades in performance with more threads. </div>
<div><br></div><div>Since running &quot;my_hybrid_app&quot; with one process behaves correctly, I suspect my linking to MPI library has some problem. Would somebody point me in the right direction? Thanks in advance.</div>
<div><br></div><div><meta http-equiv="content-type" content="text/html; charset=utf-8">Attached are the commandline used, my machine informantion and link informantion.</div><div>p.s. 1: Commandline</div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;">
<div><div><div>single process: ./my_hybrid_app &lt;number of threads&gt;</div></div></div><div><div><div>multiple process: mpirun -np 2 ./my_hybrid_app &lt;number of threads&gt;</div><meta http-equiv="content-type" content="text/html; charset=utf-8"></div>
</div><div><br></div></blockquote><div><div>p.s. 2: Machine Informantion</div></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>The first machine is CentOS 5.3 with GCC 4.1.2:</div>
</div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div><div>Target: x86_64-redhat-linux</div></div></div></blockquote><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;">
<div><div><div>Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-libgcj-multifile --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic --host=x86_64-redhat-linux</div>
</div></div></blockquote><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div><div>Thread model: posix</div></div></div></blockquote><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;">
<div><div><div>gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)</div></div></div></blockquote><div><div>The second machine is SUSE Enterprise Server 11 with GCC 4.3.4:</div></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;">
<div><div><div>Target: x86_64-suse-linux</div></div></div></blockquote><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div><div>Configured with: ../configure --prefix=/usr --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,java,ada --enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.3 --enable-ssp --disable-libssp --with-bugurl=<a href="http://bugs.opensuse.org/">http://bugs.opensuse.org/</a> --with-pkgversion=&#39;SUSE Linux&#39; --disable-libgcj --disable-libmudflap --with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit --enable-libstdcxx-allocator=new --disable-libstdcxx-pch --enable-version-specific-runtime-libs --program-suffix=-4.3 --enable-linux-futex --without-system-libunwind --with-cpu=generic --build=x86_64-suse-linux</div>
</div></div></blockquote><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div><div>Thread model: posix</div></div></div></blockquote><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;">
<div><div><div>gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux)</div></div></div></blockquote></blockquote><div><div><br></div><div>p.s. 3: ldd Informantion</div></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;">
<div><div>The first machine:</div></div><div><div><div>$ ldd my_hybrid_app</div></div></div><div><div><div>        libm.so.6 =&gt; /lib64/libm.so.6 (0x000000358d400000)</div></div></div><div><div><div>        libmpi.so.0 =&gt; /usr/local/openmpi/lib/libmpi.so.0 (0x00002af0d53a7000)</div>
</div></div><div><div><div>        libopen-rte.so.0 =&gt; /usr/local/openmpi/lib/libopen-rte.so.0 (0x00002af0d564a000)</div></div></div><div><div><div>        libopen-pal.so.0 =&gt; /usr/local/openmpi/lib/libopen-pal.so.0 (0x00002af0d5895000)</div>
</div></div><div><div><div>        libdl.so.2 =&gt; /lib64/libdl.so.2 (0x000000358d000000)</div></div></div><div><div><div>        libnsl.so.1 =&gt; /lib64/libnsl.so.1 (0x000000358f000000)</div></div></div><div><div><div>
        libutil.so.1 =&gt; /lib64/libutil.so.1 (0x000000359a600000)</div></div></div><div><div><div>        libgomp.so.1 =&gt; /usr/lib64/libgomp.so.1 (0x00002af0d5b07000)</div></div></div><div><div><div>        libpthread.so.0 =&gt; /lib64/libpthread.so.0 (0x000000358d800000)</div>
</div></div><div><div><div>        libc.so.6 =&gt; /lib64/libc.so.6 (0x000000358cc00000)</div></div></div><div><div><div>        /lib64/ld-linux-x86-64.so.2 (0x000000358c800000)</div></div></div><div><div><div>        librt.so.1 =&gt; /lib64/librt.so.1 (0x000000358dc00000)</div>
</div></div><div><div>The second machine:</div></div><div><div><div>$ ldd my_hybrid_app</div></div></div><div><div><div>        linux-vdso.so.1 =&gt;  (0x00007fff3eb5f000)</div></div></div><div><div><div>        libmpi.so.0 =&gt; /root/opt/openmpi/lib/libmpi.so.0 (0x00007f68627a1000)</div>
</div></div><div><div><div>        libm.so.6 =&gt; /lib64/libm.so.6 (0x00007f686254b000)</div></div></div><div><div><div>        libopen-rte.so.0 =&gt; /root/opt/openmpi/lib/libopen-rte.so.0 (0x00007f68622fc000)</div></div>
</div><div><div><div>        libopen-pal.so.0 =&gt; /root/opt/openmpi/lib/libopen-pal.so.0 (0x00007f68620a5000)</div></div></div><div><div><div>        libdl.so.2 =&gt; /lib64/libdl.so.2 (0x00007f6861ea1000)</div></div></div>
<div><div><div>        libnsl.so.1 =&gt; /lib64/libnsl.so.1 (0x00007f6861c89000)</div></div></div><div><div><div>        libutil.so.1 =&gt; /lib64/libutil.so.1 (0x00007f6861a86000)</div></div></div><div><div><div>        libgomp.so.1 =&gt; /usr/lib64/libgomp.so.1 (0x00007f686187d000)</div>
</div></div><div><div><div>        libpthread.so.0 =&gt; /lib64/libpthread.so.0 (0x00007f6861660000)</div></div></div><div><div><div>        libc.so.6 =&gt; /lib64/libc.so.6 (0x00007f6861302000)</div></div></div><div><div>
<div>        /lib64/ld-linux-x86-64.so.2 (0x00007f6862a58000)</div></div></div><div><div><div>        librt.so.1 =&gt; /lib64/librt.so.1 (0x00007f68610f9000)</div></div></div><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div>
I installed openmpi-1.4.2 to a user directory /root/opt/openmpi and use &quot;-L/root/opt/openmpi -Wl,-rpath,/root/opt/openmpi&quot; when linking.</div><meta http-equiv="content-type" content="text/html; charset=utf-8"><meta http-equiv="content-type" content="text/html; charset=utf-8"><meta http-equiv="content-type" content="text/html; charset=utf-8"></div>
</blockquote><div>-- <br>
<div>Huiwei Lv</div><div>PhD. student at Institute of Computing Technology, </div><div>Beijing, China</div></div><div><a href="http://asg.ict.ac.cn/lhw">http://asg.ict.ac.cn/lhw</a></div>