I&#39;ve tried to launch the application on nodes with QDR Infiniband. The first attempt with 2 processes worked, but the following was printed to the output:<div><div>[1345633953.436676] [b01:2523 :0]     mpool.c:99   MXM ERROR Invalid mempool parameter(s)</div>
<div>[1345633953.436676] [b01:2522 :0]     mpool.c:99   MXM ERROR Invalid mempool parameter(s)</div><div>--------------------------------------------------------------------------</div><div>MXM was unable to create an endpoint. Please make sure that the network link is</div>
<div>active on the node and the hardware is functioning. </div><div><br></div><div>  Error: Invalid parameter</div><div><br></div><div>--------------------------------------------------------------------------</div><div><br>
</div><div>The results from this launch didn&#39;t differ from the results of the launch without MXM.</div><div><br></div><div>Then I&#39;ve tried to launch it with 256 processes, but got the same message from each process and then the application crashed. After that I&#39;m observing the same behavior as with FDR: application hangs in the beginning. </div>
<div><br></div><div>Best regards, Pavel Mezentsev.</div><div><br></div><br><div class="gmail_quote">2012/8/22 Pavel Mezentsev <span dir="ltr">&lt;<a href="mailto:pavel.mezentsev@gmail.com" target="_blank">pavel.mezentsev@gmail.com</a>&gt;</span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello!<div><br></div><div>I&#39;ve built openmpi 1.6.1rc3 with support of MXM. But when I try to launch an application using this mtl it hangs and can&#39;t figure out why.</div>
<div><br></div><div>If I launch it with np below 128 then everything works fine since mxm isn&#39;t used. I&#39;ve tried setting the threshold to 0 and launching 2 processes with the same result: hangs on startup.</div>
<div>What could be causing this problem?</div><div><br></div><div>Here is the command I execute:</div><div><div>/opt/openmpi/1.6.1/mxm-test/bin/mpirun \</div><div>                -np $NP \</div><div>                -hostfile hosts_fdr2 \</div>

<div>                --mca mtl mxm \</div><div>                --mca btl ^tcp \</div><div>                --mca mtl_mxm_np 0 \</div><div>                -x OMP_NUM_THREADS=$NT \</div><div>                -x LD_LIBRARY_PATH \</div>

<div>                --bind-to-core \</div><div>                -npernode 16 \</div><div>                --mca coll_fca_np 0 -mca coll_fca_enable 0 \</div><div>                ./IMB-MPI1 -npmin $NP Allreduce Reduce Barrier Bcast Allgather Allgatherv</div>

</div><div><br></div><div>I&#39;m performing the tests on nodes with Intel SB processors and FDR. Openmpi was configured with the following parameters:</div><div>CC=icc CXX=icpc F77=ifort FC=ifort ./configure --prefix=/opt/openmpi/1.6.1rc3/mxm-test --with-mxm=/opt/mellanox/mxm --with-fca=/opt/mellanox/fca --with-knem=/usr/share/knem</div>

<div>I&#39;m using the latest ofed from mellanox: 1.5.3-3.1.0 on centos 6.1 with default kernel: 2.6.32-131.0.15. </div><div>The compilation with default mxm (1.0.601) failed so I installed the latest version from mellanox: 1.1.1227</div>

<div><br></div><div>Best regards, Pavel Mezentsev.</div>
</blockquote></div><br></div>