Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI users] sm btl choices
From: Ralph Castain (rhc_at_[hidden])
Date: 2010-03-01 04:51:12

Which version of OMPI are you using? We know that the 1.2 series was unreliable about removing the session directories, but 1.3 and above appear to be quite good about it. If you are having problems with the 1.3 or 1.4 series, I would definitely like to know about it.

When I was at LANL, I ran a number of tests in exactly this configuration. While the sm btl did provide some performance advantage, it wasn't very much (the bandwidth was only about 10% greater, and the latency wasn't all that different either). I set the default configuration for users to include sm as 10% isn't something to sneer at, but you could disable it without an enormous impact.

Another option would be to run an epilog that hammers the session directory. That's what LANL does, even though we didn't see much trouble with cleanup starting with the 1.3 series (still have a bunch of users stuck on 1.2). Depending on what environment you are running, you might contact folks there and get a copy of their epilog script.

On Mar 1, 2010, at 1:42 AM, David Turner wrote:

> Hi all,
> Running on a large cluster of 8-core nodes. I understand
> that the SM BTL is a "good thing". But I'm curious about
> its use of memory-mapped files. I believe these files will
> be in $TMPDIR, which defaults to /tmp.
> In our cluster, the compute nodes are stateless, so /tmp
> is actually in RAM. Keeping memory-mapped "files" in
> memory seems kind of circular, although I know little
> about these things. A bigger problem is that it appears
> OMPI does not remove the files upon completion.
> Another option is to redefine $TMPDIR to point to a
> "real" file system. In our cluster, all the available
> file systems are accessed over the IB fabric. So it
> seems that there will be IB traffic, even though the
> point of the SM BTL is to avoid this traffic.
> Given the above two constraints, might it just be
> better to disable the SM BTL entirely, and use the
> IB BTL even within a node? Of course, the "self"
> BTL should still be used if appropriate.
> Any thoughts clarifying these issues would be
> greatly appreciated. Thanks!
> --
> Best regards,
> David Turner
> User Services Group email: dpturner_at_[hidden]
> NERSC Division phone: (510) 486-4027
> Lawrence Berkeley Lab fax: (510) 486-4316
> _______________________________________________
> users mailing list
> users_at_[hidden]