We would like to set process memory limits (vmemoryuse, in csh
terms) on remote processes. Our batch system is torque/moab.
The nodes of our cluster each have 24GB of physical memory, of
which 4GB is taken up by the kernel and the root file system.
Note that these are diskless nodes, so no swap either.
We can globally set the per-process limit to 2.5GB. This works
fine if applications run "packed": 8 MPI tasks running on each
8-core node, for an aggregate limit of 20GB. However, if a job
only wants to run 4 tasks, the soft limit can safely be raised
to 5GB. 2 tasks, 10GB. 1 task, the full 20GB.
Upping the soft limit in the batch script itself only affects
the "head node" of the job. Since limits are not part of the
"environment", I can find no way propagate them to remote nodes.
If I understand how this all works, the remote processes are
started by orted, and therefore inherit its limits. Is there
any sort of orted configuration that can help here? Any other
thoughts about how to approach this?
User Services Group email: dpturner_at_[hidden]
NERSC Division phone: (510) 486-4027
Lawrence Berkeley Lab fax: (510) 486-4316