Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] too much stack size: _silently_ failback to IPoIB
From: Paul Kapinos (kapinos_at_[hidden])
Date: 2012-10-05 06:32:53


Dear Open MPI developer,
there are often problems with the user limit for the stack size (ulimit -s) on
Linux if running Fortran and/or OpenMP(=hybride) programs.

In one case we have seen the user has set the stack size in his environment by
occasion far too high - to about one TeraByte (on nodes with less than 100Gb RAM).

It turned out that Open MPI (1.6.1) cannot use InfiniBand in this environment
(cannot activate IB card / register memory / something else because of lack of
virtual memory - all memory reserved for the virtual stack?). The job seem to
fail back and run over IPoIB, according to achieved bandwidth.

The problem was that there was no single word of caution printed out, whereby
Open MPI usually warns the user iff an seemingly available high performance
network cannot be used, AFAIK. Thus the problem of the user - 15x bandwidth and
performance loss - was covered for many weeks and found only by occasion.

So, what's going wrong [if any]?

Reproducing: try to set the 'ulimit -s' in your environment to an astronomic
value, or use the attached wrapper.

$MPI_ROOT/bin/mpiexec -mca oob_tcp_if_include ib0 -mca btl_tcp_if_include ib0
-np 2 -H linuxbdc01,linuxbdc02 /home/pk224850/bin/ulimit_high.sh MPI_FastTest.exe

-- 
Dipl.-Inform. Paul Kapinos   -   High Performance Computing,
RWTH Aachen University, Center for Computing and Communication
Seffenter Weg 23,  D 52074  Aachen (Germany)
Tel: +49 241/80-24915