we see an exeedingly *virtual* memory consumtion through MPI processes
if "ulimit -s" (stack size)in profile configuration was setted higher.
Furthermore we believe, every mpi process started, wastes about the
double size of `ulimit -s` value which will be set in a fresh console
(that is, the value is configurated in e.g. .zshenv, *not* the value
actually setted in the console from which the mpiexec runs).
Sun MPI 8.2.1, an empty mpi-HelloWorld program
! either if running both processes on the same host..
.zshenv: ulimit -s 10240 --> VmPeak: 180072 kB
.zshenv: ulimit -s 102400 --> VmPeak: 364392 kB
.zshenv: ulimit -s 1024000 --> VmPeak: 2207592 kB
.zshenv: ulimit -s 2024000 --> VmPeak: 4207592 kB
.zshenv: ulimit -s 20240000 --> VmPeak: 39.7 GB!!!!
(see the attached files; the a.out binary is a mpi helloworld program
running an never ending loop).
Normally, the stack size ulimit is set to some 10 MB by us, but we see a
lot of codes which needs *a lot* of stack space, e.g. Fortran codes,
OpenMP codes (and especially fortran OpenMP codes). Users tends to
hard-code the setting-up the higher value for stack size ulimit.
Normally, the using of a lot of virtual memory is no problem, because
there is a lot of this thing :-) But... If more than one person is
allowed to work on a computer, you have to divide the ressources in such
a way that nobody can crash the box. We do not know how to limit the
real RAM used so we need to divide the RAM by means of setting virtual
memory ulimit (in our batch system e.g.. That is, for us
"virtual memory consumption" = "real memory consumption".
And real memory is not that way cheap than virtual memory.
So, why consuming the *twice* amount of stack size for each process?
And, why consuming the virtual memory at all? We guess this virtual
memory is allocated for the stack (why else it will be related to the
stack size ulimit). But, is such allocation really needed? Is there a
way to avoid the vaste of virtual memory?
Dipl.-Inform. Paul Kapinos - High Performance Computing,
RWTH Aachen University, Center for Computing and Communication
Seffenter Weg 23, D 52074 Aachen (Germany)
Tel: +49 241/80-24915