I am running into similar issues with both Mellanox and IBM HCAs.<br><br>On a node installed with RHEL6.2 and MLNX_OFED-1.5.3-3.0.0, there is a significant hit to locked memory when going with the device&#39;s max_cqe.  Here, for comparison&#39;s sake is the memory utilization for a simple MPI process when using the new cq_size default, and when restricting it to 1500:<br>
<br>cq_size = max_cqe:<br><span style="font-family:courier new,monospace">VmPeak:   348736 kB</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">VmSize:   348352 kB</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">VmLck:    292096 kB</span><br><span style="font-family:courier new,monospace">VmHWM:    304896 kB</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">VmRSS:    304896 kB</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">VmData:   333504 kB</span><br><br>cq_size = 1500<br><font face="courier new,monospace">VmPeak:    86720 kB<br>VmSize:    86336 kB<br>VmLck:     30080 kB<br>VmHWM:     42880 kB<br>
VmRSS:     42880 kB<br>VmData:    71488 kB</font><br><br>For our Power systems using the IBM eHCA, the default value exhausts memory and we can&#39;t even run.<br><br>--Brad<br><br><br><div class="gmail_quote">On Fri, Jul 6, 2012 at 5:21 AM, TERRY DONTJE <span dir="ltr">&lt;<a href="mailto:terry.dontje@oracle.com" target="_blank">terry.dontje@oracle.com</a>&gt;</span> wrote:<br>

<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000"><div>
    <br>
    <br>
    On 7/5/2012 5:47 PM, Shamis, Pavel wrote:
    <blockquote type="cite">
      <blockquote type="cite">
        <blockquote type="cite">
          <pre>I mentioned on the call that for Mellanox devices (+OFA verbs) this resource is really cheap. Do you run mellanox hca + OFA verbs ?
</pre>
        </blockquote>
        <pre>(I&#39;ll reply because I know Terry is offline for the rest of the day)

Yes, he does.
</pre>
      </blockquote>
      <pre>I asked because SUN used to have own verbs driver.
</pre>
    </blockquote></div>
    I noticed this on a Solaris machine, I am not sure I have the same
    set up for Linux but I&#39;ll look and see if I can reproduce the same
    issue on Linux.<span><font color="#888888"><br>
    <br>
    --td</font></span><div><br>
    <blockquote type="cite">
      <pre></pre>
      <blockquote type="cite">
        <pre>The heart of the question: is it incorrect to assume that we&#39;ll consume (num CQE * CQE size) registered memory for each QP opened?
</pre>
      </blockquote>
      <pre>QP or CQ ?  I think you don&#39;t want to assume anything there. Verbs (user and kernel) do their own magic there.
I think Mellanox should address this question.

Regards,
Pasha
_______________________________________________
devel mailing list
<a href="mailto:devel@open-mpi.org" target="_blank">devel@open-mpi.org</a>
<a href="http://www.open-mpi.org/mailman/listinfo.cgi/devel" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/devel</a>
</pre>
    </blockquote>
    <br>
    </div><div><div>-- <br>
      
      <div>
        
        
        <div>
          <div>
            <div>
              <div>Terry D. Dontje | Principal
                Software Engineer<br>
                <div><font color="#666666" face="Verdana">Developer
                    Tools
                    Engineering | <a href="tel:%2B1.781.442.2631" value="+17814422631" target="_blank">+1.781.442.2631</a><br>
                  </font>
                  <font color="#ff0000" face="Verdana">Oracle
                  </font><font color="#666666" face="Verdana"><b>
                      - Performance
                      Technologies</b></font><br>
                  <font color="#666666" face="Verdana">
                    95 Network Drive, Burlington, MA 01803<br>
                    Email <a href="mailto:terry.dontje@oracle.com" target="_blank">terry.dontje@oracle.com</a><br>
                  </font><br>
                </div>
              </div>
            </div>
          </div>
        </div>
      </div>
      <br>
      <br>
    </div>
  </div></div>

<br>_______________________________________________<br>
devel mailing list<br>
<a href="mailto:devel@open-mpi.org" target="_blank">devel@open-mpi.org</a><br>
<a href="http://www.open-mpi.org/mailman/listinfo.cgi/devel" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/devel</a><br></blockquote></div><br>