Skip to content
This repository has been archived by the owner on Apr 17, 2022. It is now read-only.

switch to .xz for source tarballs? #3492

Closed
wzdev-ci opened this issue May 19, 2012 · 11 comments
Closed

switch to .xz for source tarballs? #3492

wzdev-ci opened this issue May 19, 2012 · 11 comments
Labels
closed component: other normal
Milestone
unspecified

Comments

@wzdev-ci
Copy link
Contributor

resolution_fixed type_to-do (no feature requests!) | by pabs

Would it be a good idea to switch to .xz for source tarballs? Debian can handle those now and they provide better compression than both gzip and bzip2. 7-zip can handle these on Windows and MacOS.

Issue migrated from trac:3492 at 2022-04-16 09:49:54 -0700
@wzdev-ci
Copy link
Contributor Author

wzdev-ci commented Jun 2, 2012

cybersphinx changed _comment0 which not transferred by tractive

@wzdev-ci
Copy link
Contributor Author

wzdev-ci commented Jun 2, 2012

cybersphinx commented

$ pv -p -a -b warzone2100-bugfixes.tar | gzip > warzone2100-bugfixes.tar.gz
 318MB 0:00:14 [21.8MB/s]
$ pv -p -a -b warzone2100-bugfixes.tar | gzip --best > warzone2100-bugfixes.tar.gz--best
 318MB 0:00:29 [10.9MB/s]
$ pv -p -a -b warzone2100-bugfixes.tar | xz > warzone2100-bugfixes.tar.xz
 318MB 0:02:50 [1.86MB/s]
$ pv -p -a -b warzone2100-bugfixes.tar | xz -e > warzone2100-bugfixes.tar.xz-e
 318MB 0:03:43 [1.42MB/s]
$ cat warzone2100-bugfixes.tar | lrzip | pv -a -b -t > warzone2100-bugfixes.tar.lrzip
Average Compression Speed:  7.925MB/s.
 318MB 0:00:39 [1.57MB/s]

$ pv -p -a -b warzone2100-bugfixes.tar.gz | gunzip > /dev/null
 191MB 0:00:04 [45.1MB/s]
$ pv -p -a -b warzone2100-bugfixes.tar.gz--best | gunzip > /dev/null
 191MB 0:00:02 [71.9MB/s]
$ pv -p -a -b warzone2100-bugfixes.tar.xz | unxz > /dev/null
 171MB 0:00:09 [  19MB/s]
$ pv -p -a -b warzone2100-bugfixes.tar.xz-e | unxz > /dev/null
 171MB 0:00:08 [19.6MB/s]
$ cat warzone2100-bugfixes.tar.lrzip | lrunzip | pv -a -b -t > /dev/null
Average DeCompression Speed: 45.286MB/s
 62.3MB 0:00:07 [43.9MB/s]

From my point of view, the only alternative to gzip is lrzip, everything else is just too slow for the little size advantage it offers over gzip. (Run on a 4 x 3 GHz Athlon II. Output slightly changed to be readable, especially the lrzip sizes are exchanged because of the position of pv in the pipe. Size in the first block is the source size, second is the resulting archive. Yes, lrzip saves ~66% in contrast to the ~11% of xz.)

Edit: My second point is that the stupid "supercompression" options the autotools use gain us exactly nothing (gunzip is so fast that cache-cold actually matters, on several runs there is no difference extracting the gz and gz --best variants.)

@wzdev-ci
Copy link
Contributor Author

wzdev-ci commented Jun 2, 2012

Per changed status from new to closed

@wzdev-ci
Copy link
Contributor Author

wzdev-ci commented Jun 2, 2012

Per changed resolution from `` to closed

@wzdev-ci
Copy link
Contributor Author

wzdev-ci commented Jun 2, 2012

Per commented

Feature request, not a bug.

@wzdev-ci wzdev-ci closed this as completed Jun 2, 2012
@wzdev-ci
Copy link
Contributor Author

wzdev-ci commented Jun 10, 2012 •
edited

cybersphinx commented

Replying to Warzone2100/old-trac-import#3492 (comment:1):

Yes, lrzip saves ~66% in contrast to the ~11% of xz.

... and I think I know why now. For some reason, the tarball make dist gave me included the game data three times, lrzip recognized that and could reduce it to the actually included information. (Didn't really pay attention to the file sizes then...)

So, redone with the beta10 tarball:

$ time sh -c "cat warzone2100-3.1_beta10.tar | gzip | pv -p -a -b > warzone2100-3.1_beta10.tar.gz"
  67MB [15.1MB/s]
real    0m4.475s
$ time sh -c "cat warzone2100-3.1_beta10.tar | gzip --best | pv -p -a -b > warzone2100-3.1_beta10.tar.gz--best"
66.9MB [8.67MB/s]
real    0m7.769s
$ time sh -c "cat warzone2100-3.1_beta10.tar | xz | pv -p -a -b > warzone2100-3.1_beta10.tar.xz"
61.2MB [1.05MB/s]
real    0m58.471s
$ time sh -c "cat warzone2100-3.1_beta10.tar | xz -e | pv -p -a -b > warzone2100-3.1_beta10.tar.xz-e"
61.1MB [ 752kB/s]
real    1m23.251s
$ time sh -c "cat warzone2100-3.1_beta10.tar | lrzip | pv -p -a -b > warzone2100-3.1_beta10.tar.lrzip"
61.2MB [3.13MB/s]
real    0m19.584s

$ du -s *|sort -n
62576   warzone2100-3.1_beta10.tar.xz-e
62680   warzone2100-3.1_beta10.tar.xz
62716   warzone2100-3.1_beta10.tar.lrzip
68492   warzone2100-3.1_beta10.tar.gz--best
68632   warzone2100-3.1_beta10.tar.gz
117260  warzone2100-3.1_beta10.tar

Well, lrzip still wins in the speed department against xz, and --best/-e are still pointless.

@wzdev-ci
Copy link
Contributor Author

cybersphinx uploaded file 0001-Default-to-xz-compression-for-make-dist.patch (0.9 KiB)

@wzdev-ci
Copy link
Contributor Author

cybersphinx changed resolution from closed to fixed

@wzdev-ci
Copy link
Contributor Author

cybersphinx commented

Default to xz compression for make dist.

Also set xz options to the xz default -6, since the automake default -e
is noticeably slower without noticeably better compression. Fixes #3492.

  • Changeset: [/changeset/512ef27f4d4185d004977e0be4b972bfa130c57d 512ef27f4d4185d004977e0be4b972bfa130c57d]
  • URL: [512]ef27f4d4185d004977e0be4b972bfa130c57d

2 similar comments
@wzdev-ci
Copy link
Contributor Author

cybersphinx commented

Default to xz compression for make dist.

Also set xz options to the xz default -6, since the automake default -e
is noticeably slower without noticeably better compression. Fixes #3492.

  • Changeset: [/changeset/512ef27f4d4185d004977e0be4b972bfa130c57d 512ef27f4d4185d004977e0be4b972bfa130c57d]
  • URL: [512]ef27f4d4185d004977e0be4b972bfa130c57d

@wzdev-ci
Copy link
Contributor Author

cybersphinx commented

Default to xz compression for make dist.

Also set xz options to the xz default -6, since the automake default -e
is noticeably slower without noticeably better compression. Fixes #3492.

  • Changeset: [/changeset/512ef27f4d4185d004977e0be4b972bfa130c57d 512ef27f4d4185d004977e0be4b972bfa130c57d]
  • URL: [512]ef27f4d4185d004977e0be4b972bfa130c57d

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
closed component: other normal
Projects
None yet
Milestone
unspecified
Development

No branches or pull requests
1 participant
@wzdev-ci