People can correct me if I am wrong, but are you NOT suppose to run applications directly and only suppose to submit them using the batch process "qsub" to prevent all this overuse of resources?
> Date: Thu, 19 Nov 2009 19:21:46 +0100 > From: firstname.lastname@example.org > To: email@example.com > Subject: [OMPI users] exceedingly virtual memory consumption of MPI environment if higher-setting "ulimit -s" > > Hi volks, > > we see an exeedingly *virtual* memory consumtion through MPI processes > if "ulimit -s" (stack size)in profile configuration was setted higher. > > Furthermore we believe, every mpi process started, wastes about the > double size of `ulimit -s` value which will be set in a fresh console > (that is, the value is configurated in e.g. .zshenv, *not* the value > actually setted in the console from which the mpiexec runs). > > Sun MPI 8.2.1, an empty mpi-HelloWorld program > ! either if running both processes on the same host.. > > .zshenv: ulimit -s 10240 --> VmPeak: 180072 kB > .zshenv: ulimit -s 102400 --> VmPeak: 364392 kB > .zshenv: ulimit -s 1024000 --> VmPeak: 2207592 kB > .zshenv: ulimit -s 2024000 --> VmPeak: 4207592 kB > .zshenv: ulimit -s 20240000 --> VmPeak: 39.7 GB!!!! > (see the attached files; the a.out binary is a mpi helloworld program > running an never ending loop). > > > > Normally, the stack size ulimit is set to some 10 MB by us, but we see a > lot of codes which needs *a lot* of stack space, e.g. Fortran codes, > OpenMP codes (and especially fortran OpenMP codes). Users tends to > hard-code the setting-up the higher value for stack size ulimit. > > Normally, the using of a lot of virtual memory is no problem, because > there is a lot of this thing :-) But... If more than one person is > allowed to work on a computer, you have to divide the ressources in such > a way that nobody can crash the box. We do not know how to limit the > real RAM used so we need to divide the RAM by means of setting virtual > memory ulimit (in our batch system e.g.. That is, for us > "virtual memory consumption" = "real memory consumption". > And real memory is not that way cheap than virtual memory. > > > So, why consuming the *twice* amount of stack size for each process? > > And, why consuming the virtual memory at all? We guess this virtual > memory is allocated for the stack (why else it will be related to the > stack size ulimit). But, is such allocation really needed? Is there a > way to avoid the vaste of virtual memory? > > best regards, > Paul Kapinos > > > > > > > > > > > > -- > Dipl.-Inform. Paul Kapinos - High Performance Computing, > RWTH Aachen University, Center for Computing and Communication > Seffenter Weg 23, D 52074 Aachen (Germany) > Tel: +49 241/80-24915