Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] OpenMPI job initializing problem
From: Ralph Castain (rhc_at_[hidden])
Date: 2014-03-21 09:05:53


One thing to check would be the time spent between MPI_Init and MPI_Finalize - i.e., see if the time difference is caused by differences in init and finalize themselves. My guess is that is the source - would help us target the problem.

On Mar 20, 2014, at 9:00 PM, Beichuan Yan <beichuan.yan_at_[hidden]> wrote:

> Here is an example of my data measured in seconds:
>
> communication overhead = commuT + migraT + print, compuT is computational cost, totalT = compuT + communication overhead, overhead% denotes percentage of communication overhead
>
> intelmpi (walltime=00:03:51)
> iter [commuT migraT printT] compuT totalT overhead%
> 3999 4.945993e-03 2.689362e-04 1.440048e-04 1.689100e-02 2.224994e-02 2.343795e+01
> 5999 4.938126e-03 1.451969e-04 2.689362e-04 1.663089e-02 2.198315e-02 2.312373e+01
> 7999 4.904985e-03 1.490116e-04 1.451969e-04 1.678491e-02 2.198410e-02 2.298933e+01
> 9999 4.915953e-03 1.380444e-04 1.490116e-04 1.687193e-02 2.207494e-02 2.289473e+01
>
> openmpi (walltime=00:04:32)
> iter [commuT migraT printT] compuT totalT overhead%
> 3999 3.574133e-03 1.139641e-04 1.089573e-04 1.598001e-02 1.977706e-02 1.864836e+01
> 5999 3.574848e-03 1.189709e-04 1.139641e-04 1.599526e-02 1.980305e-02 1.865278e+01
> 7999 3.571033e-03 1.168251e-04 1.189709e-04 1.601100e-02 1.981783e-02 1.860879e+01
> 9999 3.587008e-03 1.258850e-04 1.168251e-04 1.596618e-02 1.979589e-02 1.875587e+01
>
> It can be seen that Open MPI is faster in both communication and computation measured by MPI_Wtime calls, but the wall time reported by PBS pro is larger.
>
>
> -----Original Message-----
> From: users [mailto:users-bounces_at_[hidden]] On Behalf Of Gus Correa
> Sent: Thursday, March 20, 2014 15:08
> To: Open MPI Users
> Subject: Re: [OMPI users] OpenMPI job initializing problem
>
> On 03/20/2014 04:48 PM, Beichuan Yan wrote:
>> Ralph and Noam,
>>
>> Thanks for the clarifications, they are important.
> I could be wrong in understanding the filesystem.
>>
>> Spirit appears to use a scratch directory for
> shared memory backing which is mounted on Lustre, and does not seem to have local directories or does not allow user to change TEMPDIR. Here is the info:
>> [compute node]$ stat -f -L -c %T /tmp
>> tmpfs
>> [compute node]$ stat -f -L -c %T /home/yanb/scratch lustre
>>
>
> So, /tmp is a tmpfs, in memory/RAM.
> Maybe they don't open writing permissions for regular users on /tmp?
>
>> On another university supercomputer, I found the following:
>> node0448[~]$ stat -f -L -c %T /tmp
>> ramfs
>> node0448[~]$ stat -f -L -c %T /home/yanb/scratch/ lustre Is this /tmp
>> at compute node a local directory? I don't know how to tell it.
>>
>> Thanks,
>> Beichuan
>>
>>
>>
>> -----Original Message-----
>> From: users [mailto:users-bounces_at_[hidden]] On Behalf Of Ralph
>> Castain
>> Sent: Thursday, March 20, 2014 12:13
>> To: Open MPI Users
>> Subject: Re: [OMPI users] OpenMPI job initializing problem
>>
>>
>> On Mar 20, 2014, at 9:48 AM, Beichuan Yan <beichuan.yan_at_[hidden]> wrote:
>>
>>> Hi,
>>>
>>> Today I tested OMPI v1.7.5rc5 and surprisingly, it works like a charm!
>>>
>>> I found discussions related to this issue:
>>>
>>> 1. http://www.open-mpi.org/community/lists/users/2011/11/17688.php
>>> The correct solution here is get your sys admin to make /tmp local. Making /tmp NFS mounted across multiple nodes is a major "faux pas" in the Linux world - it should never be done, for the reasons stated by Jeff.
>>>
>>> my comment: for most clusters I have used, /tmp is NOT local. Open MPI community may not enforce it.
>>
>> We don't enforce anything, but /tmp being network mounted is a VERY
>> unusual situation in the cluster world, and highly unrecommended
>>
>>
>>>
>>> 2. http://www.open-mpi.org/community/lists/users/2011/11/17684.php
>>> In the upcoming OMPI v1.7, we revamped the shared memory setup code such that it'll actually use /dev/shm properly, or use some other mechanism other than a mmap file backed in a real filesystem. So the issue goes away.
>>>
>>> my comment: up to OMPI v1.7.4, this shmem issue is still there. However, it is resolved in OMPI v1.7.5rc5. This is surprising.
>>>
>>> Anyway, OMPI v1.7.5rc5 works well for multi-processes-on-one-node (shmem) mode on Spirit. There is no need to tune TCP or IB parameters to use it. My code just runs well:
>>>
>>> My test data takes 20 minutes to run with OMPI v1.7.4, but needs less than 1 minute with OMPI v1.7.5rc5. I don't know what the magic is. I am wondering when OMPI v1.7.5 final will be released.
>>>
>>> I will update performance comparison between Intel MPI and Open MPI.
>>>
>>> Thanks,
>>> Beichuan
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: users [mailto:users-bounces_at_[hidden]] On Behalf Of Gus
>>> Correa
>>> Sent: Friday, March 07, 2014 18:41
>>> To: Open MPI Users
>>> Subject: Re: [OMPI users] OpenMPI job initializing problem
>>>
>>> On 03/06/2014 04:52 PM, Beichuan Yan wrote:
>>>> No, I did all these and none worked.
>>>>
>>>> I just found, with exact the same code, data and job settings, a job can really run one day while cannot the other day. It is NOT repeatable. I don't know what the problem is: hardware? OpenMPI? PBS Pro?
>>>>
>>>> Anyway, I may have to give up using OpenMPI on that system and switch to IntelMPI which always work.
>>>>
>>>> Thanks,
>>>> Beichuan
>>>
>>> Well, this machine may have been setup to run only Intel MPI (DAPL?) and SGI MPI.
>>> It is a pity that it doesn't seem to work with OpenMPI.
>>>
>>> In any case, good luck with your research project.
>>>
>>> Gus Correa
>>>
>>>>
>>>> -----Original Message-----
>>>> From: users [mailto:users-bounces_at_[hidden]] On Behalf Of Gus
>>>> Correa
>>>> Sent: Thursday, March 06, 2014 13:51
>>>> To: Open MPI Users
>>>> Subject: Re: [OMPI users] OpenMPI job initializing problem
>>>>
>>>> On 03/06/2014 03:35 PM, Beichuan Yan wrote:
>>>>> Gus,
>>>>>
>>>>> Yes, 10.148.0.0/16 is the IB subnet.
>>>>>
>>>>> I did try others but none worked:
>>>>> #export
>>>>> TCP="--mca btl sm,openib"
>>>>> No run, no output
>>>>
>>>> If I remember right, and unless this changed in recent OMPI vervsions, you also need "self":
>>>>
>>>> -mca btl sm,openib,self
>>>>
>>>> Alternatively, you could rule out tcp:
>>>>
>>>> -mca btl ^tcp
>>>>
>>>>>
>>>>> #export
>>>>> TCP="--mca btl sm,openib --mca btl_tcp_if_include 10.148.0.0/16"
>>>>> No run, no output
>>>>>
>>>>> Beichuan
>>>>
>>>> Likewise, "self" is missing here.
>>>>
>>>> Also, I don't know if you can ask for openib and also add --mca btl_tcp_if_include 10.148.0.0/16.
>>>> Note that one turns off tcp (I think), whereas the other requests a
>>>> tcp interface (or that the IB interface with IPoIB functionality).
>>>> That combination sounds weird to me.
>>>> The OMPI developers may clarify if this is valid syntax/syntax combination.
>>>>
>>>> I would try simply -mca btl sm,openib,self, which is likely to give
>>>> you the IB transport with verbs, plus shared memory intra-node, plus
>>>> the
>>>> (mandatory?) self (loopback interface?).
>>>> In my experience, this will also help identify any malfunctioning IB HCA in the nodes (with a failure/error message).
>>>>
>>>>
>>>> I hope it helps,
>>>> Gus Correa
>>>>
>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: users [mailto:users-bounces_at_[hidden]] On Behalf Of Gus
>>>>> Correa
>>>>> Sent: Thursday, March 06, 2014 13:16
>>>>> To: Open MPI Users
>>>>> Subject: Re: [OMPI users] OpenMPI job initializing problem
>>>>>
>>>>> Hi Beichuan
>>>>>
>>>>> So, it looks like that now the program runs, even though with specific settings depending on whether you're using OMPI 1.6.5 or 1.7.4, right?
>>>>>
>>>>> It looks like the problem now is performance, right?
>>>>>
>>>>> System load affects performance, but unless the network is overwhelmed, or perhaps the Lustre file system is hanging or too slow, I would think that a walltime increase from 1min to 10min is not related to system load, but something else.
>>>>>
>>>>> Do you remember the setup that gave you 1min walltime?
>>>>> Was it the same that you sent below?
>>>>> Do you happen to know which nodes?
>>>>> Are you sharing nodes with other jobs, or are you running alone on the nodes?
>>>>> Sharing with other processes may slow down your job.
>>>>> If you request all cores in the node, PBS should give you a full node (unless they tricked PBS to think the nodes have more cores than they actually do).
>>>>> How do you request the nodes in your #PBS directives?
>>>>> Do you request nodes and ppn, or do you request procs?
>>>>>
>>>>> I suggest that you do:
>>>>> cat $PBS_NODEFILE
>>>>> in your PBS script, just to document which nodes are actually given to you.
>>>>>
>>>>> Also helpful to document/troubleshoot is to add -v and -tag-output to your mpiexec command line.
>>>>>
>>>>>
>>>>> The difference in walltime could be due to some malfunction of IB HCAs on the nodes, for instance.
>>>>> Since you are allowing (if I remember right) the use of TCP, OpenMPI will try to use any interfaces that you did not rule out.
>>>>> If your mpiexec command line doesn't make any restriction, it will use anything available, if I remember right.
>>>>> (Jeff will correct me in the next second.) If your mpiexec command line has mca btl_tcp_if_include 10.148.0.0/16 it will use the 10.148.0.0/16 subnet in with TCP transport, I think.
>>>>> (Jeff will cut my list subscription after that one, for spreading
>>>>> misinformation.)
>>>>>
>>>>> In either case my impression is that you may have left a door open to the use of non-IB (and non-IB-verbs) transport.
>>>>>
>>>>> Is 10.148.0.0/16 the an Infiniband subnet or an Ethernet subnet?
>>>>>
>>>>> Did you remeber Jeff's suggestion from a while ago to avoid TCP (over Ethernet or over IB), and stick to IB verbs?
>>>>>
>>>>>
>>>>> Is 10.148.0.0/16 the IB or the Ethernet subnet?
>>>>>
>>>>> On 03/02/2014 02:38 PM, Jeff Squyres (jsquyres) wrote:
>>>>>> Both 1.6.x and 1.7.x/1.8.x will need verbs.h to use the native
>>>>>> verbs network stack.
>>>>>>
>>>>>> You can use emulated TCP over IB (e.g., using the OMPI TCP BTL),
>>>>>> but it's nowhere near as fast/efficient the native verbs network stack.
>>>>>>
>>>>>
>>>>>
>>>>> You could force the use of IB verbs with
>>>>>
>>>>> -mca btl ^tcp
>>>>>
>>>>> or with
>>>>>
>>>>> -mca btl sm,openib,self
>>>>>
>>>>> on the mpiexec command line.
>>>>>
>>>>> In this case, if any of the IB HCAs on the nodes is bad, the job
>>>>> will abort with an error message, instead of running too slow (if
>>>>> it is using other networks).
>>>>>
>>>>> There are also ways to tell OMPI to do a more verbose output, that
>>>>> may perhaps help diagnose the problem.
>>>>> ompi_info | grep verbose
>>>>> may give some hints (I confess I don't remember them).
>>>>>
>>>>>
>>>>> Believe me, this did happen to me, i.e., to run MPI programs in a
>>>>> cluster that had all sorts of non-homogeneous nodes, some with
>>>>> faulty IB HCAs, some with incomplete OFED installation, some that
>>>>> were not mounting shared file systems properly, etc.
>>>>> [I didn't administer that one!]
>>>>> Hopefully that is not the problem you are facing, but verbose
>>>>> output may help anyways.
>>>>>
>>>>> I hope this helps,
>>>>> Gus Correa
>>>>>
>>>>>
>>>>>
>>>>> On 03/06/2014 01:49 PM, Beichuan Yan wrote:
>>>>>> 1. For $TMPDIR and $TCP, there are four combinations by commenting on/off (note the system's default TMPDIR=/work3/yanb):
>>>>>> export TMPDIR=/work1/home/yanb/tmp TCP="--mca btl_tcp_if_include
>>>>>> 10.148.0.0/16"
>>>>>>
>>>>>> 2. I tested the 4 combinations for OpenMPI 1.6.5 and OpenMPI 1.7.4 respectively for the pure-MPI mode (no OPENMP threads; 8 nodes, each node runs 16 processes). The results are weird: of all 8 cases, only TWO of them can run, but run so slow:
>>>>>>
>>>>>> OpenMPI 1.6.5:
>>>>>> export TMPDIR=/work1/home/yanb/tmp TCP="--mca btl_tcp_if_include
>>>>>> 10.148.0.0/16"
>>>>>> Warning: shared-memory, /work1/home/yanb/tmp/ Run, take 10
>>>>>> minutes, slow
>>>>>>
>>>>>> OpenMPI 1.7.4:
>>>>>> #export TMPDIR=/work1/home/yanb/tmp #TCP="--mca btl_tcp_if_include
>>>>>> 10.148.0.0/16"
>>>>>> Warning: shared-memory /work3/yanb/605832.SPIRIT/ Run, take 10
>>>>>> minutess, slow
>>>>>>
>>>>>> So you see, a) openmpi 1.6.5 and 1.7.4 need different settings to
>>>>>> run;
>>>>> b) whether specifying TMPDIR, I got the shared memory warning.
>>>>>>
>>>>>> 3. But a few days ago, OpenMPI 1.6.5 worked great and took only 1
>>>>>> minute
>>>>> (now it takes 10 minutes). I am so confused by the results.
>>>>> Does the system loading level or fluctuation or PBS pro affect
>>>>> OpenMPI performance?
>>>>>>
>>>>>> Thanks,
>>>>>> Beichuan
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: users [mailto:users-bounces_at_[hidden]] On Behalf Of Gus
>>>>>> Correa
>>>>>> Sent: Tuesday, March 04, 2014 08:48
>>>>>> To: Open MPI Users
>>>>>> Subject: Re: [OMPI users] OpenMPI job initializing problem
>>>>>>
>>>>>> Hi Beichuan
>>>>>>
>>>>>> So, from "df" it looks like /home is /work1, right?
>>>>>>
>>>>>> Also, "mount" shows only /work[1-4], not the other
>>>>>> 7 CWFS panfs (Panasas?), which apparently are not available in the compute nodes/blades.
>>>>>>
>>>>>> I presume you have access and are using only some of the
>>>>>> /work[1-4]
>>>>>> (lustre) file systems for all your MPI and other software installation, right? Not the panfs, right?
>>>>>>
>>>>>> Awkward that it doesn't work, because lustre is supposed to be a parallel file system, highly available to all nodes (assuming it is mounted on all nodes).
>>>>>>
>>>>>> It also shows a small /tmp with a tmpfs file system, which is volatile, in memory:
>>>>>>
>>>>>> http://en.wikipedia.org/wiki/Tmpfs
>>>>>>
>>>>>> I would guess they don't let you write there, so TMPDIR=/tmp may not be a possible option, but this is just a wild guess.
>>>>>> Or maybe OMPI requires an actual non-volatile file system to write its shared memory auxiliary files and other stuff that normally goes on /tmp? [Jeff, Ralph, help!!] I kind of remember some old discussion on this list about this, but maybe it was in another list.
>>>>>>
>>>>>> [You could ask the sys admin about this, and perhaps what he
>>>>>> recommends to use to replace /tmp.]
>>>>>>
>>>>>> Just in case they may have some file system mount point mixup, you could try perhaps TMPDIR=/work1/yanb/tmp (rather than /home) You could also try TMPDIR=/work3/yanb/tmp, as if I remember right this is another file system you have access to (not sure anymore, it may have been in the previous emails).
>>>>>> Either way, you may need to create the tmp directory beforehand.
>>>>>>
>>>>>> **
>>>>>>
>>>>>> Any chances that this is an environment mixup?
>>>>>>
>>>>>> Say, that you may be inadvertently using the SGI-MPI mpiexec Using a /full/path/to/mpiexec in your job may clarify this.
>>>>>>
>>>>>> "which mpiexec" will tell, but since the environment on the compute nodes may not be exactly the same as in the login node, it may not be reliable information.
>>>>>>
>>>>>> Or perhaps you may not be pointing to the OMPI libraries?
>>>>>> Are you exporting PATH and LD_LIBRARY_PATH on .bashrc/.tcshrc, with the OMPI items (bin and lib) *PREPENDED* (not appended), so as to take precedence over other possible/SGI/pre-existent MPI items?
>>>>>>
>>>>>> Those are pretty (ugly) common problems.
>>>>>>
>>>>>> **
>>>>>>
>>>>>> I hope this helps,
>>>>>> Gus Correa
>>>>>>
>>>>>> On 03/03/2014 10:13 PM, Beichuan Yan wrote:
>>>>>>> 1. info from a compute node
>>>>>>> -bash-4.1$ hostname
>>>>>>> r32i1n1
>>>>>>> -bash-4.1$ df -h /home
>>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>> 10.148.18.45_at_o2ib:10.148.18.46_at_o2ib:/fs1
>>>>>>> 1.2P 136T 1.1P 12% /work1 -bash-4.1$
>>>>>>> mount devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on
>>>>>>> /tmp type tmpfs (rw,size=150m) none on /proc/sys/fs/binfmt_misc
>>>>>>> type binfmt_misc
>>>>>>> (rw) cpuset on /dev/cpuset type cpuset (rw)
>>>>>>> 10.148.18.45_at_o2ib:10.148.18.46_at_o2ib:/fs1 on /work1 type lustre
>>>>>>> (rw,flock)
>>>>>>> 10.148.18.76_at_o2ib:10.148.18.164_at_o2ib:/fs2 on /work2 type lustre
>>>>>>> (rw,flock)
>>>>>>> 10.148.18.104_at_o2ib:10.148.18.165_at_o2ib:/fs3 on /work3 type lustre
>>>>>>> (rw,flock)
>>>>>>> 10.148.18.132_at_o2ib:10.148.18.133_at_o2ib:/fs4 on /work4 type lustre
>>>>>>> (rw,flock)
>>>>>>>
>>>>>>>
>>>>>>> 2. For "export TMPDIR=/home/yanb/tmp", I created it beforehand, and I did see mpi-related temporary files there when the job gets started.
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: users [mailto:users-bounces_at_[hidden]] On Behalf Of Gus
>>>>>>> Correa
>>>>>>> Sent: Monday, March 03, 2014 18:23
>>>>>>> To: Open MPI Users
>>>>>>> Subject: Re: [OMPI users] OpenMPI job initializing problem
>>>>>>>
>>>>>>> Hi Beichuan
>>>>>>>
>>>>>>> OK, it says "unclassified.html", so I presume it is not a problem.
>>>>>>>
>>>>>>> The web site says the computer is an SGI ICE X.
>>>>>>> I am not familiar to it, so what follows are guesses.
>>>>>>>
>>>>>>> The SGI site brochure suggests that the nodes/blades have local disks:
>>>>>>> https://www.sgi.com/pdfs/4330.pdf
>>>>>>>
>>>>>>> The file systems prefixed with IP addresses (work[1-4]) and with panfs (cwfs and CWFS[1-6]) and a colon (:) are shared exports (not local), but not necessarily NFS (panfs may be Panasas?).
>>>>>>> From this output it is hard to tell where /home is, but I would guess it is also shared (not local).
>>>>>>> Maybe "df -h /home" will tell. Or perhaps "mount".
>>>>>>>
>>>>>>> You may be logged in to a login/service node, so although it does have a /tmp (your ls / shows tmp), this doesn't guarantee that the compute nodes/blades also do.
>>>>>>>
>>>>>>> Since your jobs failed when you specified TMPDIR=/tmp, I would guess /tmp doesn't exist on the nodes/blades, or is not writable.
>>>>>>>
>>>>>>> Did you try to submit a job with, say, "mpiexec -np 16 ls -ld /tmp"?
>>>>>>> This should tell if /tmp exists on the nodes, if it is writable.
>>>>>>>
>>>>>>> A stupid question:
>>>>>>> When you tried your job with this:
>>>>>>>
>>>>>>> export TMPDIR=/home/yanb/tmp
>>>>>>>
>>>>>>> Did you create the directory /home/yanb/tmp beforehand?
>>>>>>>
>>>>>>> Anyway, you may need to ask the help of a system administrator of this machine.
>>>>>>>
>>>>>>> Gus Correa
>>>>>>>
>>>>>>> On 03/03/2014 07:43 PM, Beichuan Yan wrote:
>>>>>>>> Gus,
>>>>>>>>
>>>>>>>> I am using this system: http://centers.hpc.mil/systems/unclassified.html#Spirit. I don't know exactly configurations of the file system. Here is the output of "df -h":
>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>>> /dev/sda6 919G 16G 857G 2% /
>>>>>>>> tmpfs 32G 0 32G 0% /dev/shm
>>>>>>>> /dev/sda5 139M 33M 100M 25% /boot
>>>>>>>> adfs3v-s:/adfs3/hafs14
>>>>>>>> 6.5T 678G 5.5T 11% /scratch
>>>>>>>> adfs3v-s:/adfs3/hafs16
>>>>>>>> 6.5T 678G 5.5T 11% /var/spool/mail
>>>>>>>> 10.148.18.45_at_o2ib:10.148.18.46_at_o2ib:/fs1
>>>>>>>> 1.2P 136T 1.1P 12% /work1
>>>>>>>> 10.148.18.132_at_o2ib:10.148.18.133_at_o2ib:/fs4
>>>>>>>> 1.2P 793T 368T 69% /work4
>>>>>>>> 10.148.18.104_at_o2ib:10.148.18.165_at_o2ib:/fs3
>>>>>>>> 1.2P 509T 652T 44% /work3
>>>>>>>> 10.148.18.76_at_o2ib:10.148.18.164_at_o2ib:/fs2
>>>>>>>> 1.2P 521T 640T 45% /work2
>>>>>>>> panfs://172.16.0.10/CWFS
>>>>>>>> 728T 286T 443T 40% /p/cwfs
>>>>>>>> panfs://172.16.1.61/CWFS1
>>>>>>>> 728T 286T 443T 40% /p/CWFS1
>>>>>>>> panfs://172.16.0.210/CWFS2
>>>>>>>> 728T 286T 443T 40% /p/CWFS2
>>>>>>>> panfs://172.16.1.125/CWFS3
>>>>>>>> 728T 286T 443T 40% /p/CWFS3
>>>>>>>> panfs://172.16.1.224/CWFS4
>>>>>>>> 728T 286T 443T 40% /p/CWFS4
>>>>>>>> panfs://172.16.1.224/CWFS5
>>>>>>>> 728T 286T 443T 40% /p/CWFS5
>>>>>>>> panfs://172.16.1.224/CWFS6
>>>>>>>> 728T 286T 443T 40% /p/CWFS6
>>>>>>>> panfs://172.16.1.224/CWFS7
>>>>>>>> 728T 286T 443T 40% /p/CWFS7
>>>>>>>>
>>>>>>>> 1. My home directory is /home/yanb.
>>>>>>>> My simulation files are located at /work3/yanb.
>>>>>>>> The default TMPDIR set by system is just /work3/yanb
>>>>>>>>
>>>>>>>> 2. I did try not to set TMPDIR and let it default, which is just case 1 and case 2.
>>>>>>>> Case1: #export TMPDIR=/home/yanb/tmp
>>>>>>>> TCP="--mca btl_tcp_if_include 10.148.0.0/16"
>>>>>>>> It gives no apparent reason.
>>>>>>>> Case2: #export TMPDIR=/home/yanb/tmp
>>>>>>>> #TCP="--mca btl_tcp_if_include 10.148.0.0/16"
>>>>>>>> It gives warning of shared memory file on network file system.
>>>>>>>>
>>>>>>>> 3. With "export TMPDIR=/tmp", the job gives the same, no apparent reason.
>>>>>>>>
>>>>>>>> 4. FYI, "ls /" gives:
>>>>>>>> ELT apps cgroup hafs1 hafs12 hafs2 hafs5 hafs8 home lost+found mnt p root selinux tftpboot var work3
>>>>>>>> admin bin dev hafs10 hafs13 hafs3 hafs6 hafs9 lib media net panfs sbin srv tmp work1 work4
>>>>>>>> app boot etc hafs11 hafs15 hafs4 hafs7 hafs_x86_64 lib64 misc opt proc scratch sys usr work2 workspace
>>>>>>>>
>>>>>>>> Beichuan
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: users [mailto:users-bounces_at_[hidden]] On Behalf Of Gus
>>>>>>>> Correa
>>>>>>>> Sent: Monday, March 03, 2014 17:24
>>>>>>>> To: Open MPI Users
>>>>>>>> Subject: Re: [OMPI users] OpenMPI job initializing problem
>>>>>>>>
>>>>>>>> Hi Beichuan
>>>>>>>>
>>>>>>>> If you are using the university cluster, chances are that /home is not local, but on an NFS share, or perhaps Lustre (which you may have mentioned before, I don't remember).
>>>>>>>>
>>>>>>>> Maybe "df -h" will show what is local what is not.
>>>>>>>> It works for NFS, it prefixes file systems with the server name, but I don't know about Lustre.
>>>>>>>>
>>>>>>>> Did you try just not to set TMPDIR and let it default?
>>>>>>>> If the default TMPDIR is on Lustre (did you say this?, anyway I
>>>>>>>> don't
>>>>>>>> remember) you could perhaps try to force it to /tmp:
>>>>>>>> export TMPDIR=/tmp,
>>>>>>>> If the cluster nodes are diskfull /tmp is likely to exist and be local to the cluster nodes.
>>>>>>>> [But the cluster nodes may be diskless ... :( ]
>>>>>>>>
>>>>>>>> I hope this helps,
>>>>>>>> Gus Correa
>>>>>>>>
>>>>>>>> On 03/03/2014 07:10 PM, Beichuan Yan wrote:
>>>>>>>>> How to set TMPDIR to a local filesystem? Is /home/yanb/tmp a local filesystem? I don't know how to tell a directory is local file system or network file system.
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: users [mailto:users-bounces_at_[hidden]] On Behalf Of
>>>>>>>>> Jeff Squyres (jsquyres)
>>>>>>>>> Sent: Monday, March 03, 2014 16:57
>>>>>>>>> To: Open MPI Users
>>>>>>>>> Subject: Re: [OMPI users] OpenMPI job initializing problem
>>>>>>>>>
>>>>>>>>> How about setting TMPDIR to a local filesystem?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mar 3, 2014, at 3:43 PM, Beichuan Yan<beichuan.yan_at_[hidden]> wrote:
>>>>>>>>>
>>>>>>>>>> I agree there are two cases for pure-MPI mode: 1. Job fails with no apparent reason; 2 job complains shared-memory file on network file system, which can be resolved by " export TMPDIR=/home/yanb/tmp", /home/yanb/tmp is my local directory. The default TMPDIR points to a Lustre directory.
>>>>>>>>>>
>>>>>>>>>> There is no any other output. I checked my job with "qstat -n" and found that processes were actually not started on compute nodes even though PBS Pro has "started" my job.
>>>>>>>>>>
>>>>>>>>>> Beichuan
>>>>>>>>>>
>>>>>>>>>>> 3. Then I test pure-MPI mode: OPENMP is turned off, and each compute node runs 16 processes (clearly shared-memory of MPI is used). Four combinations of "TMPDIR" and "TCP" are tested:
>>>>>>>>>>> case 1:
>>>>>>>>>>> #export TMPDIR=/home/yanb/tmp TCP="--mca btl_tcp_if_include
>>>>>>>>>>> 10.148.0.0/16"
>>>>>>>>>>> mpirun $TCP -np 64 -npernode 16 -hostfile $PBS_NODEFILE
>>>>>>>>>>> ./paraEllip3d input.txt
>>>>>>>>>>> output:
>>>>>>>>>>> Start Prologue v2.5 Mon Mar 3 15:47:16 EST 2014 End Prologue
>>>>>>>>>>> v2.5 Mon Mar 3 15:47:16 EST 2014
>>>>>>>>>>> -bash: line 1: 448597 Terminated /var/spool/PBS/mom_priv/jobs/602244.service12.SC
>>>>>>>>>>> Start Epilogue v2.5 Mon Mar 3 15:50:51 EST 2014 Statistics
>>>>>>>>>>> cpupercent=0,cput=00:00:00,mem=7028kb,ncpus=128,vmem=495768kb
>>>>>>>>>>> ,
>>>>>>>>>>> w
>>>>>>>>>>> all
>>>>>>>>>>> t
>>>>>>>>>>> i
>>>>>>>>>>> m
>>>>>>>>>>> e
>>>>>>>>>>> =00:03:24 End Epilogue v2.5 Mon Mar 3 15:50:52 EST 2014
>>>>>>>>>>
>>>>>>>>>> It looks like you have two general cases:
>>>>>>>>>>
>>>>>>>>>> 1. The job fails for no apparent reason (like above), or 2.
>>>>>>>>>> The job complains that your TMPDIR is on a shared filesystem
>>>>>>>>>>
>>>>>>>>>> Right?
>>>>>>>>>>
>>>>>>>>>> I think the real issue, then, is to figure out why your jobs are failing with no output.
>>>>>>>>>>
>>>>>>>>>> Is there anything in the stderr output?
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Jeff Squyres
>>>>>>>>>> jsquyres_at_[hidden]
>>>>>>>>>> For corporate legal information go to:
>>>>>>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> users mailing list
>>>>>>>>>> users_at_[hidden]
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>> _______________________________________________
>>>>>>>>>> users mailing list
>>>>>>>>>> users_at_[hidden]
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jeff Squyres
>>>>>>>>> jsquyres_at_[hidden]
>>>>>>>>> For corporate legal information go to:
>>>>>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> users_at_[hidden]
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>> _______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> users_at_[hidden]
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users