I am runing open-mpi 1.1.1-1 compiled from OFED1.1 which I downloaded
from their website.
I am using SGE installed via OSCAR 5.0 and when running under SGE I
get the "mca_mpool_openib_register: ibv_reg_mr(0x590000,528384) failed
with error: Cannot allocate memory" error discussed at length in your
When I run from the command line using mpirun, I don't get the errors.
Of course, I don't know how to tell if the code is actually using the
IB interface instead of the GigE network...
I tried the suggestions in the FAQ regarding setting the memlock
parameter in /etc/security/limits.conf: and all the nodes return
"unlimited" in response to "ulimit -l" after rebooting the nodes. The
problem persists under SGE and still does not appear when simply using
I assumed it would work since openmpi 1.1.1 was included as working
with SGE in OSCAR 5.0, but I don't know how different that version and
the one included with OFED is.
Any suggestions would be appreciated.