Subject: Re: [MTT users] Patch to add --local-scratch option
From: Ethan Mallove (ethan.mallove_at_[hidden])
Date: 2008-09-16 16:26:49
On Tue, Sep/16/2008 12:07:31PM, Ethan Mallove wrote:
> Nice! A few thoughts ...
>
> What happens if one uses --local-scratch, but leaves out the
> --scratch option? In this case, I think MTT should assume
> --scratch is the same as --local-scratch.
>
> Could the "local" in --local-scratch ever be misleading?
> Couldn't a user ever use a mounted filesystem that's faster
> than all their local filesystem? Should it be
> --fast-scratch?
>
> For future work, how about --scratch taking a (CSV?) list of
> scratch directories. MTT then does a quick check for which
> is the fastest filesystem (if such a check is
> possible/feasible), and proceeds accordingly. That is, doing
> everything it possible can in a fast scratch (builds,
> writing out metadata, etc.), and installing the MPI(s) into
> the slow mounted scratch. Would this be possible?
>
> On Mon, Sep/15/2008 06:15:43PM, Tim Mattox wrote:
> > Hello,
> > Attached is a patchfile for the mtt trunk that adds a
> > --local-scratch <dir_name>
> > option to client/mtt. You can also specify something like
> > this in your [MTT] ini section:
> > local_scratch = &shell("echo /tmp/`whoami`_mtt")
> >
> > This local-scratch directory is then used for part of the --mpi-install
> > phase to speed up your run. Specifically, the source-code of the
> > MPI is untarred there, configure is run, make all, and make check.
> > Then, when make install is invoked the MPI is installed into the
> > usual place as if you hadn't used --local-scratch. If you don't
> > use --local-scratch, then the builds occur in the usual place that
> > they have before.
> >
> > For the clusters at IU that seem to have slow NSF home directories,
> > this cuts the --mpi-install phase time in half.
> >
> > The downside is that if the MPI build fails, your build directory is out
> > on some compile-node's /tmp and is harder to go debug. But, since
> > mpi build failures are now rare, this should make for quicker turnaround
> > for the general case.
> >
> > I think I adjusted the code properly for the vpath build case, but I've
> > never used that so haven't tested it. Also, I adjusted the free disk space
> > check code. Right now it only checks the free space on --scratch,
> > and won't detect if --local-scratch is full. If people really care, I
> > could make it check both later. But for now, if your /tmp is full
> > you probably have other problems to worry about.
> >
> > Comments? Can you try it out, and if I get no objections, I'd like
> > to put this into the MTT trunk this week.
>
> I'll test your patch soon.
>
I was able to do an MPI get/install after applying the patch.
I ran:
$ client/mtt --local-scratch /tmp/foo --scratch /nfs/bar ...
Which resulted in the build directory ending up in:
/tmp/foo/.../src
And the resulting installation ending up in the below
directory via the DESTDIR macro passed to "make install":
/nfs/bar/.../src
The patch seems to do what it's supposed to do.
-Ethan
> -Ethan
>
>
> > --
> > Tim Mattox, Ph.D. - http://homepage.mac.com/tmattox/
> > tmattox_at_[hidden] || timattox_at_[hidden]
> > I'm a bright... http://www.the-brights.net/
> > _______________________________________________
> > mtt-users mailing list
> > mtt-users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>