process 0 starts test Epsilon = 0.0000000010 process 2 starts test process 3 starts test process 4 starts test process 1 starts test process 5 starts test ####### process 2 yields partial sum: 108000000.4 size, block: 1080000000 1080000001 MPI_Reduce in einem! ####### process 0 yields partial sum: 108000000.4 ####### process 5 yields partial sum: 108000000.4 ####### process 3 yields partial sum: 108000000.4 size, block: 1080000000 1080000001 MPI_Reduce in einem! size, block: 1080000000 1080000001 MPI_Reduce in einem! ####### process 1 yields partial sum: 108000000.4 size, block: 1080000000 1080000001 MPI_Reduce in einem! size, block: 1080000000 1080000001 MPI_Reduce in einem! ####### process 4 yields partial sum: 108000000.4 size, block: 1080000000 1080000001 MPI_Reduce in einem! -------------------------------------------------------------------------- A process failed to create a queue pair. This usually means either the device has run out of queue pairs (too many connections) or there are insufficient resources available to allocate a queue pair (out of memory). The latter can happen if either 1) insufficient memory is available, or 2) no more physical memory can be registered with the device. For more information on memory registration see the Open MPI FAQs at: http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages Local host: linuxbmc0372.rz.RWTH-Aachen.DE Local device: mlx4_0 Queue pair type: Reliable connected (RC) -------------------------------------------------------------------------- [linuxbmc0372.rz.RWTH-Aachen.DE:12898] *** An error occurred in MPI_Barrier [linuxbmc0372.rz.RWTH-Aachen.DE:12898] *** on communicator MPI_COMM_WORLD [linuxbmc0372.rz.RWTH-Aachen.DE:12898] *** MPI_ERR_OTHER: known error not in list [linuxbmc0372.rz.RWTH-Aachen.DE:12898] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort rank: 2 VmPeak: 25775548 kB rank: 4 VmPeak: 25775552 kB rank: 3 VmPeak: 25775664 kB rank: 0 VmPeak: 17338008 kB -------------------------------------------------------------------------- mpiexec has exited due to process rank 2 with PID 12897 on node linuxbmc0372 exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpiexec (as reported here). -------------------------------------------------------------------------- rank: 1 VmPeak: 25775792 kB rank: 5 VmPeak: 17104304 kB Failure executing command /opt/MPI/openmpi-1.6.1rc2/linux/intel/bin/mpiexec -x LD_LIBRARY_PATH -x PATH -x OMP_NUM_THREADS -x MPI_NAME -mca oob_tcp_if_include ib0 -mca btl_tcp_if_include ib0 --hostfile /tmp/pk224850/cluster_52665/hostfile-39693 -np 6 memusage a.out 1080000000 1080000001