I wasn't advocating against having the epilogue per se, but was more curious if there was some issue going on that we did not know about.  If there isn't an issue then great.

--td

On 11/4/2011 9:59 AM, Ralph Castain wrote:
That isn't the situation, Terry. We had problems with early OMPI releases, particularly the 1.2 series. In response, the labs wrote an epilogue to ensure that the session directories were removed. Executing the epilogue is now standard operating procedure, even though our more recent releases do a much better job of cleanup.

Frankly, it's a good idea anyway. It hurts nothing, takes milliseconds to do, and guarantees nothing got left behind (e.g., if someone was using a debug version of OMPI and directed opal_output to a file).

On Nov 4, 2011, at 4:43 AM, TERRY DONTJE wrote:

David, are you saying your jobs consistently leave behind session files after the job exits?  It really shouldn't even in the case when a job aborts, I thought, mpirun took great pains to cleanup after itself.    Can you tell us what version of OMPI you are running with?  I think I could see kill -9 of mpirun and processes below would cause turds to be left behind.

--td

On 11/4/2011 2:37 AM, David Turner wrote:
% df /tmp
Filesystem           1K-blocks      Used Available Use% Mounted on
-                     12330084    822848  11507236   7% /
% df /
Filesystem           1K-blocks      Used Available Use% Mounted on
-                     12330084    822848  11507236   7% /

That works out to 11GB.  But...

The compute nodes have 24GB.  Freshly booted, about 3.2GB is
consumed by the kernel, various services, and the root file system.
At this time, usage of /tmp is essentially nil.

We set user memory limits to 20GB.

I would imagine that the size of the session directories depends on a
number of factors; perhaps the developers can comment on that.  I have
only seen total sizes in the 10s of MBs on our 8-node, 24GB nodes.

As long as they're removed after each job, they don't really compete
with the application for available memory.

On 11/3/11 8:40 PM, Ed Blosch wrote:
Thanks very much, exactly what I wanted to hear. How big is /tmp?

-----Original Message-----
From: users-bounces@open-mpi.org [mailto:users-bounces@open-mpi.org] On
Behalf Of David Turner
Sent: Thursday, November 03, 2011 6:36 PM
To: users@open-mpi.org
Subject: Re: [OMPI users] EXTERNAL: Re: How to set up state-less node /tmp
for OpenMPI usage

I'm not a systems guy, but I'll pitch in anyway.  On our cluster,
all the compute nodes are completely diskless.  The root file system,
including /tmp, resides in memory (ramdisk).  OpenMPI puts these
session directories therein.  All our jobs run through a batch
system (torque).  At the conclusion of each batch job, an epilogue
process runs that removes all files belonging to the owner of the
current batch job from /tmp (and also looks for and kills orphan
processes belonging to the user).  This epilogue had to written
by our systems staff.

I believe this is a fairly common configuration for diskless
clusters.

On 11/3/11 4:09 PM, Blosch, Edwin L wrote:
Thanks for the help.  A couple follow-up-questions, maybe this starts to
go outside OpenMPI:

What's wrong with using /dev/shm?  I think you said earlier in this thread
that this was not a safe place.

If the NFS-mount point is moved from /tmp to /work, would a /tmp magically
appear in the filesystem for a stateless node?  How big would it be, given
that there is no local disk, right?  That may be something I have to ask the
vendor, which I've tried, but they don't quite seem to get the question.

Thanks




-----Original Message-----
From: users-bounces@open-mpi.org [mailto:users-bounces@open-mpi.org] On
Behalf Of Ralph Castain
Sent: Thursday, November 03, 2011 5:22 PM
To: Open MPI Users
Subject: Re: [OMPI users] EXTERNAL: Re: How to set up state-less node /tmp
for OpenMPI usage


On Nov 3, 2011, at 2:55 PM, Blosch, Edwin L wrote:

I might be missing something here. Is there a side-effect or performance
loss if you don't use the sm btl?  Why would it exist if there is a wholly
equivalent alternative?  What happens to traffic that is intended for
another process on the same node?

There is a definite performance impact, and we wouldn't recommend doing
what Eugene suggested if you care about performance.

The correct solution here is get your sys admin to make /tmp local. Making
/tmp NFS mounted across multiple nodes is a major "faux pas" in the Linux
world - it should never be done, for the reasons stated by Jeff.



Thanks


-----Original Message-----
From: users-bounces@open-mpi.org [mailto:users-bounces@open-mpi.org] On
Behalf Of Eugene Loh
Sent: Thursday, November 03, 2011 1:23 PM
To: users@open-mpi.org
Subject: Re: [OMPI users] EXTERNAL: Re: How to set up state-less node
/tmp for OpenMPI usage

Right.  Actually "--mca btl ^sm".  (Was missing "btl".)

On 11/3/2011 11:19 AM, Blosch, Edwin L wrote:
I don't tell OpenMPI what BTLs to use. The default uses sm and puts a
session file on /tmp, which is NFS-mounted and thus not a good choice.

Are you suggesting something like --mca ^sm?


-----Original Message-----
From: users-bounces@open-mpi.org [mailto:users-bounces@open-mpi.org] On
Behalf Of Eugene Loh
Sent: Thursday, November 03, 2011 12:54 PM
To: users@open-mpi.org
Subject: Re: [OMPI users] EXTERNAL: Re: How to set up state-less node
/tmp for OpenMPI usage

I've not been following closely.  Why must one use shared-memory
communications?  How about using other BTLs in a "loopback" fashion?
_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users





--
<Mail Attachment.gif>
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle - Performance Technologies
95 Network Drive, Burlington, MA 01803
Email terry.dontje@oracle.com



_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________ users mailing list users@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users

--
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle - Performance Technologies
95 Network Drive, Burlington, MA 01803
Email terry.dontje@oracle.com