On Apr 16, 2007, at 6:48 PM, Adams, Brian M wrote:
> I am attempting to port Sandia's DAKOTA code from MVAPICH to the
> OpenMPI/Intel environment on Sandia's thunderbird cluster. I can
> successfully build DAKOTA in the default tbird software
> environment, but
> I'm having runtime problems when DAKOTA attempts to make a system
> Typical output looks like:
> from an64 to: an64 error polling HP CQ with status LOCAL LENGTH ERROR
> status number 1 for wr_id 5714048 opcode 0
Unfortunately, making calls to system() or fork() will fail when
using the OFED 1.1 stack (such as on thunderbird). The fun part is
that the failure is not immediate; calling fork() or system() will
cause odd/interesting errors later in your program (such as what you
The only way around this is to call fork()/system() before the call
to MPI_INIT or after the call to MPI_FINALIZE.
The OFED 1.2 stack has proper support for fork()/system(), but I
don't know what tbird's plans are for upgrading (I doubt it has been
discussed yet since OFED 1.2 is still going through its release
process -- it's not final yet).
> Note: Both programs run fine with MVAPICH on tbird,
This is probably luck; I wouldn't count on it happening reliably.
> and with OpenMPI or
> MPICH on my Linux x86_64 SMP workstation.
There are many environments where fork() and system() work fine
(e.g., when only using tcp and shared memory), but the OFED 1.1 stack
is unfortunately not one of them.
I wish I had a better answer for you, but I don't. Sorry!