Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: [hwloc-devel] tarball growing
From: Brice Goglin (Brice.Goglin_at_[hidden])
Date: 2010-09-28 02:27:55


The bz2 tarball of hwloc 1.0.2 was 2.1MB. hwloc 1.1 will be at least
2.7MB. I know that bandwidth is free, but I am still not confortable
with the size increasing that much.

Obviously, the problem comes from tarballs under tests/linux:

605774 28 sept. 08:12 tests/linux/256ppc-8n8s4t.tar.gz
311293 28 sept. 08:12 tests/linux/256ppc-8n8s4t-nocache.tar.gz
232021 28 sept. 08:12 tests/linux/256ppc-8n8s4t-nosys.tar.gz
 81856 28 sept. 08:12 tests/linux/48amd64-4d2n6c-sparse.tar.gz
 72906 28 sept. 08:12 tests/linux/96em64t-4n4d3ca2co.tar.gz
 72839 28 sept. 08:12 tests/linux/128ia64-17n4s2c.tar.gz
 53033 28 sept. 08:12 tests/linux/64fake-4n2s2ca2c2t.tar.gz
 41148 28 sept. 08:12 tests/linux/256ia64-64n2s2c.tar.gz
 29786 28 sept. 08:12 tests/linux/8em64t-4c2t.tar.gz
 23947 28 sept. 08:12 tests/linux/256ia64-64n2s2c.tar.gz.output
 20391 28 sept. 08:12 tests/linux/256ppc-8n8s4t-nocache.tar.gz.output
 20391 28 sept. 08:12 tests/linux/256ppc-8n8s4t.tar.gz.output

Ideas:
1) Remove useless tarballs: Some early machines are very simple and
likely covered by larger machines added later to the test suite. But
those early tarballs are small anyway, there's not much improvement to
expect there.
2) The *extracted* size of the tarballs looks mostly related to
directory sizes (multiple of 4kB), I don't know if we can hope an
improvement inside the tarball for these, I assume that tar already
compresses them properly...
3) Reduce duplication between variants: Have the test script remove some
parts of the extracted tarball directly instead of keeping multiple
variants of the same machines with/without some files. This one should
help, I'll work on it tonight.
4) Remove some useless files from the tarballs. Assuming we know what
we'll need in the future... there's likely something to remove under the
topology, cache, and device-tree directories.
5) Stop distributing tarballs. having them in SVN is useful. I am not
sure about this one, maybe wait for some users to actually complain
about the tarball size :)
6) Try to generate some of our huge tarballs manually. Any intern to do
this? :)

Any other idea?

Brice