Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] tarball growing
From: Jeff Squyres (jsquyres) (jsquyres_at_[hidden])
Date: 2010-09-28 06:40:14


Eh. Other than not liking it, is there a *problem* with the tarball getting larger?

We could also make 2 tarballs if you really care - one with the tests and one without.

Sent from my PDA. No type good.

On Sep 28, 2010, at 2:28 AM, "Brice Goglin" <Brice.Goglin_at_[hidden]> wrote:

> The bz2 tarball of hwloc 1.0.2 was 2.1MB. hwloc 1.1 will be at least
> 2.7MB. I know that bandwidth is free, but I am still not confortable
> with the size increasing that much.
>
> Obviously, the problem comes from tarballs under tests/linux:
>
> 605774 28 sept. 08:12 tests/linux/256ppc-8n8s4t.tar.gz
> 311293 28 sept. 08:12 tests/linux/256ppc-8n8s4t-nocache.tar.gz
> 232021 28 sept. 08:12 tests/linux/256ppc-8n8s4t-nosys.tar.gz
> 81856 28 sept. 08:12 tests/linux/48amd64-4d2n6c-sparse.tar.gz
> 72906 28 sept. 08:12 tests/linux/96em64t-4n4d3ca2co.tar.gz
> 72839 28 sept. 08:12 tests/linux/128ia64-17n4s2c.tar.gz
> 53033 28 sept. 08:12 tests/linux/64fake-4n2s2ca2c2t.tar.gz
> 41148 28 sept. 08:12 tests/linux/256ia64-64n2s2c.tar.gz
> 29786 28 sept. 08:12 tests/linux/8em64t-4c2t.tar.gz
> 23947 28 sept. 08:12 tests/linux/256ia64-64n2s2c.tar.gz.output
> 20391 28 sept. 08:12 tests/linux/256ppc-8n8s4t-nocache.tar.gz.output
> 20391 28 sept. 08:12 tests/linux/256ppc-8n8s4t.tar.gz.output
>
>
>
> Ideas:
> 1) Remove useless tarballs: Some early machines are very simple and
> likely covered by larger machines added later to the test suite. But
> those early tarballs are small anyway, there's not much improvement to
> expect there.
> 2) The *extracted* size of the tarballs looks mostly related to
> directory sizes (multiple of 4kB), I don't know if we can hope an
> improvement inside the tarball for these, I assume that tar already
> compresses them properly...
> 3) Reduce duplication between variants: Have the test script remove some
> parts of the extracted tarball directly instead of keeping multiple
> variants of the same machines with/without some files. This one should
> help, I'll work on it tonight.
> 4) Remove some useless files from the tarballs. Assuming we know what
> we'll need in the future... there's likely something to remove under the
> topology, cache, and device-tree directories.
> 5) Stop distributing tarballs. having them in SVN is useful. I am not
> sure about this one, maybe wait for some users to actually complain
> about the tarball size :)
> 6) Try to generate some of our huge tarballs manually. Any intern to do
> this? :)
>
> Any other idea?
>
> Brice
>
> _______________________________________________
> hwloc-devel mailing list
> hwloc-devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel