I have an OpenMPI program compiled with a version of OpenMPI built using the ifort 10.1
compiler. I can compile and run this code with no problem, using the 32 bit
version of ifort. And I can also submit batch jobs using torque with this 32-bit code.
However, compiling the same code to produce a 64 bit executable produces a code
that runs correctly only in the simplest cases. It does not run correctly when run
under the torque batch queuing system, running for awhile and then giving a
segmentation violation in s section of code that is fine in the 32 bit version.
I have to run the mpi multinode jobs using our torque batch queuing system,
but we do have the capability of running the jobs in an interactive batch environment.
If I do a qsub -I -l nodes=1:x4gb
I get an interactive session on the remote node assigned to my job. I can run the
job using either
mpirun -np 1 ./MPI_li_64
and the job runs successfully to completion. I can also
start an interactive shell using
qsub -I -l nodes=1:ppn=2:x4gb
and I will get a single dual processor (or greater node). On this single node,
mpirun -np 2 ./MPI_li_64 works.
However, if instead I ask for two nodes in my interactive batch node,
qsub -I -l nodes=2:x4gb,
Two nodes will be assigned to me but when I enter
mpirun -np 2 ./MPI_li_64
the job runs awhile, then fails with a
mpirun noticed that process rank 1 with PID 23104 on node n339 exited on signal 11 (Segmentation fault).
I can trace this in the intel debugger and see that the segmentation fault is occuring in what should
be good code, and in code that executes with no problem when everything is compiled 32-bit. I am
at a loss for what could be preventing this code to run within the batch queuing environment in the