Subject: Re: [MTT users] Patch to add --local-scratch option
From: Ethan Mallove (ethan.mallove_at_[hidden])
Date: 2008-10-31 13:42:28
On Thu, Oct/30/2008 06:04:06PM, Jeff Squyres wrote:
> Tim and I discussed this on the phone the other day and then I talked about
> it more with Ethan. After all this discussion, I committed a variant of
> Tim's patch into an HG for review before putting it back on the SVN trunk:
>
> http://www.open-mpi.org/hg/hgwebdir.cgi/jsquyres/mtt-fast-scratch/
>
> If you have a milliways account, you can get to it via ssh and therefore be
> able to push back into it. http clones won't be able to push back.
>
> Here's the changes I made:
>
> 1. I changed Tim's original concept a little bit: instead of having a
> "local" scratch, I called it "fast scratch". The idea is that there is a
> "fast" scratch space *that neither persistent nor global*. It gets whacked
> (by default) at the end of the MTT invocation. It has no protection for
> multiple MTT invocations using it simultaneously. Any part of the MTT code
> base can use this fast scratch if they want to. If somewhere wants to use
> it, they are responsible for making a "safe" section in the fast scratch
> tree. The only place currently using it is the MPI::Install phase (it
> makes $fast_scratch/mpi-install/ and does stuff under there); if there's a
> build portion of that install, the fast scratch can be used. As Tim set
> it, the builddir is set to point into the fast scratch.
>
> 2. New [MTT] section fields:
>
> - delete_fast_scratch: defaults to 1, but can be overridden. If 1, the
> entire fast scratch tree will be rm -rf'ed when the current INI file run is
> complete.
>
> - save_fast_scratch_files: a comma-delimited list of files to be saved from
> the fast scratch tree before it is whacked. It defaults to "config.log".
> A Files:: subroutine (currently only invoked by MPI::Install.pm) scans a
> designated sub-tree in the fast scratch and sees if it finds any files
> matching those names (e.g., config.log). If it does, it preserves the
> directory hierarchy it found in the fast scratch when copying the file to
> the persistent scratch tree. For example, saving config.log in an OMPI
> build, you'll end up with the following in $scratch/installs/<4CHAR>/fssf
> (= fast scratch save files):
>
> ./src/openmpi-1.4a1r19857/ompi/contrib/vt/vt/config.log
> ./src/openmpi-1.4a1r19857/ompi/contrib/vt/vt/extlib/otf/config.log
> ./src/openmpi-1.4a1r19857/ompi/mca/io/romio/romio/config.log
> ./src/openmpi-1.4a1r19857/config.log
>
> 3. I toyed with the idea of adding an option for saving the *entire* fast
> scratch MPI install tree to the permanent scratch (on the argument that
> [effectively] tar'ing up the entire fast scratch and writing it to the
> regular scratch would still be faster than doing all the interactive IO to
> build OMPI on the regular scratch), but I have run out of time today and
> therefore probably won't get to it. :-)
>
> This could probably be featurized a bit more, but I figured that this would
> be helpful to several of us and would be worth reviewing and getting into
> the SVN trunk, even if we lack a few features.
>
> What do you guys think?
I just ran this, and it seemed to work fine:
$ client/mtt --fast-scratch /tmp/mtt-jeff-fast-scratch \
--scratch /workspace/mtt-jeff-slow-scratch
-Ethan
>
>
> On Sep 19, 2008, at 3:44 PM, Jeff Squyres wrote:
>
>> Excellent points.
>>
>> What about a slightly different approach that would allow us to be exactly
>> specific?
>>
>> --scratch=dir1[,dir2[,dir3[,...]]]
>>
>> And then in the INI file, have fields that indicate which scratch dir you
>> want them to use. For example, the fact that the OMPI MPI Install plugin
>> does a build is really a side-effect (not all MPI Install plugins do
>> builds). So we could have a field:
>>
>> ompi_build_scratch = &scratch(1)
>>
>> Or, heck, it doesn't even need to be a function of --scratch at all.
>> "ompi_build_scratch = <foo>" alone could be good enough.
>>
>>
>>
>> On Sep 19, 2008, at 12:43 PM, Tim Mattox wrote:
>>
>>> I've also been thinking about this a bit more, and although
>>> having the name match the INI section name has some appeal,
>>> I ultimately think the best name is: --mpi-build-scratch, since
>>> that is what it does. As Ethan mentioned, the actual MPI
>>> install goes into --scratch. And on the other side of it,
>>> the MPI Get also goes into --scratch. The --mpi-build scratch
>>> is only used for untaring/copying the MPI source tree, running
>>> config, make, and make check. The actual "make install"
>>> simply copies the binaries from --mpi-build-scratch into --scratch.
>>>
>>> As for names like local-scratch or fast-scratch, they don't convey
>>> what it's used for, so should it be fast-for-big-files, of
>>> fast-for-small-files?
>>> Or similarly, "local" to my cluster, my node, or what?
>>> I think mpi-build-scratch conveys the most useful meaning, since you
>>> should pick a filesystem that is tuned (or at least not horrible) for
>>> doing configure/make.
>>>
>>> Unfortunately, I won't have time today to get the patch adjusted
>>> and into svn. Maybe on Monday.
>>>
>>> On Fri, Sep 19, 2008 at 11:23 AM, Ethan Mallove <ethan.mallove_at_[hidden]>
>>> wrote:
>>>> On Thu, Sep/18/2008 05:35:13PM, Jeff Squyres wrote:
>>>>> On Sep 18, 2008, at 10:45 AM, Ethan Mallove wrote:
>>>>>
>>>>>>> Ah, yeah, ok, now I see why you wouldl call it --mpi-install-scratch,
>>>>>>> so
>>>>>>> that it matches the MTT ini section name. Sure, that works for me.
>>>>>>
>>>>>> Since this does seem like a feature that should eventually
>>>>>> propogate to all the other phases (except for Test run),
>>>>>> what will we call the option to group all the fast phase
>>>>>> scratches?
>>>>>
>>>>> --scratch
>>>>>
>>>>> :-)
>>>>>
>>>>> Seriously, *if* we ever implement the other per-phase scratches, I
>>>>> think
>>>>> having one overall --scratch and fine-grained per-phase specifications
>>>>> fine. I don't think we need to go overboard to have a way to say I
>>>>> want
>>>>> phases X, Y, and Z to use scratch A. Meaning that you could just use
>>>>> --X-scratch=A --Y-scratch=A and --Z-scratch=A.
>>>>
>>>> --mpi-install-scratch actually has MTT install (using
>>>> DESTDIR) into --scratch. Is that confusing? Though
>>>> --fast-scratch could also be misleading, as I could see a
>>>> user thinking that --fast-scratch will do some magical
>>>> optimization to make their NFS directory go faster. I guess
>>>> I'm done splitting hairs over --mpi-install-scratch :-)
>>>>
>>>> -Ethan
>>>>
>>>>
>>>>>
>>>>> --
>>>>> Jeff Squyres
>>>>> Cisco Systems
>>>>>
>>>>> _______________________________________________
>>>>> mtt-users mailing list
>>>>> mtt-users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>> _______________________________________________
>>>> mtt-users mailing list
>>>> mtt-users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>
>>>
>>>
>>>
>>> --
>>> Tim Mattox, Ph.D. - http://homepage.mac.com/tmattox/
>>> tmattox_at_[hidden] || timattox_at_[hidden]
>>> I'm a bright... http://www.the-brights.net/
>>> _______________________________________________
>>> mtt-users mailing list
>>> mtt-users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>
>>
>> --
>> Jeff Squyres
>> Cisco Systems
>>
>> _______________________________________________
>> mtt-users mailing list
>> mtt-users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>
>
> --
> Jeff Squyres
> Cisco Systems
>
> _______________________________________________
> mtt-users mailing list
> mtt-users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users