Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] Are there ways to reduce the memory used by OpenMPI?
From: Blosch, Edwin L (edwin.l.blosch_at_[hidden])
Date: 2009-10-01 14:56:18

Are there are tuning parameters than I can use to reduce the amount of memory used by OpenMPI? I would very much like to use OpenMPI instead of MVAPICH, but I'm on a cluster where memory usage is the most important consideration. Here are three results which capture the problem:

With the "leave_pinned" behavior turned on, I get good performance (19.528, lower is better)

mpirun --prefix /usr/mpi/intel/openmpi-1.2.8 --machinefile /var/spool/torque/aux/7972.fwnaeglingio -np 28 --mca btl ^tcp --mca mpi_leave_pinned 1 --mca mpool_base_use_mem_hooks 1 -x LD_LIBRARY_PATH -x MPI_ENVIRONMENT=1 /tmp/7972.fwnaeglingio/falconv4_ibm_openmpi -cycles 100 -ri restart.0 -ro /tmp/7972.fwnaeglingio/restart.0
Compute rate (processor-microseconds/cell/cycle): 19.528
Total memory usage: 38155.3477 MB (38.1553 GB)

Turning off the leave_pinned behavior, I get considerably slower performance (28.788), but the memory usage is unchanged (still 38 GB)

mpirun --prefix /usr/mpi/intel/openmpi-1.2.8 --machinefile /var/spool/torque/aux/7972.fwnaeglingio -np 28 -x LD_LIBRARY_PATH -x MPI_ENVIRONMENT=1 /tmp/7972.fwnaeglingio/falconv4_ibm_openmpi -cycles 100 -ri restart.0 -ro /tmp/7972.fwnaeglingio/restart.0
Compute rate (processor-microseconds/cell/cycle): 28.788
Total memory usage: 38335.7656 MB (38.3358 GB)

Using MVAPICH, the performance is in the middle (23.6), but the memory usage is reduced by 5 to 6 GB out of 38 GB, a significant decrease to me.

/usr/mpi/intel/mvapich-1.1.0/bin/mpirun_rsh -ssh -np 28 -hostfile /var/spool/torque/aux/7972.fwnaeglingio LD_LIBRARY_PATH="/usr/mpi/intel/mvapich-1.1.0/lib/shared:/usr/mpi/intel/openmpi-1.2.8/lib64:/appserv/intel/fce/10.1.008/lib:/appserv/intel/cce/10.1.008/lib" MPI_ENVIRONMENT=1 /tmp/7972.fwnaeglingio/falconv4_ibm_mvapich -cycles 100 -ri restart.0 -ro /tmp/7972.fwnaeglingio/restart.0
Compute rate (processor-microseconds/cell/cycle): 23.608
Total memory usage: 32753.0586 MB (32.7531 GB)

I didn't see anything in the FAQ that discusses memory usage other than the impact of the "leave_pinned" option, which apparently does not affect the memory usage in my case. But I figure there must be a justification why OpenMPI would use 6 GB more than MVAPICH on the same case.

Thanks for any insights. Also attached is the output of ompi_info -a.