It seems as if the problem stems from /tmp being mounted as an NFS space that is shared between the compute nodes.
This problem can be resolved in a variety of ways. Below are a few avenues that can help get around the "globally mounted /tmp space" issue, but others are welcome to add to the list.
o Change the place where ORTE stores its session information
-mca orte_tmpdir_base /path/to/some/local/store
-mca orte_tmpdir_base /dev/shm
**Note: the following options are only available in Open MPI v1.5.5+**
o Change where shmem mmap places its files.
-mca shmem_mmap_relocate_backing_file -1 -mca shmem_mmap_backing_file_base_dir /dev/shm
o Change the backing facility used by the sm mpool and sm BTL to posix or sysv
-mca shmem posix
-mca shmem sysv
On Apr 24, 2012, at 12:34 PM, Seyyed Mohtadin Hashemi wrote: