Thanks a lot for your replies.
I am looking into upgrading MPI to a newer version. As I use a few custom
built libraries as part of my main parallel application that recommend the
use of 1.1.2, I first need to check compatibility issues with the newer
version before I can upgrade.
The following is the output of ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 268288
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 8192
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 268288
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
As I mentioned before, I am able to run it successfully with 997 processes
around 6 out of 10 times with the rest failing. I tried with 500 processes
and I do get the odd failure in that case too.