Skip to content
zlib-ngPublic

About

Whole-process gzip CLI benchmarks across implementations

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

41 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

gzipbench

Whole-process gzip command line benchmarks across implementations.

bench.py times whole processes, stdin to stdout, so startup, threading, and IO all count. Each run covers the level ladder, thread sweeps at level 6, a decompression grid crossing decoders with producers, and a deflate block census.

Tools

Variant Tool Parallel flag Requires
gzip-ng -p gzip-ng -p
pigz -p pigz -p
pigzpp -p pigzpp (-E zlib) -p
bgzip -@ bgzip (htslib) -@
migz MiGz (pool of cpus) always a JDK
gzip-ng gzip-ng serial
minigzip zlib-ng's minigzip
gzip gzip

Tools in tools/bin are preferred, then PATH, missing ones are skipped and noted. GZIPNG, PIGZ, PIGZPP, GZIP, MINIGZIP, BGZIP, and JAVA override the lookup. MiGz has no standalone binary, tools/fetch_migz.py pulls the pinned jars from Maven Central once and compiles a small pipe CLI, JVM start stays part of its measurements.

tools/build_tools.py builds gzip-ng, minigzip, pigz, pigzpp, and bgzip against zlib-ng develop into tools/bin, so every zlib-linked tool measures the same library. It writes tools/zlibng.json, and the graphs put that zlib-ng version and commit in their footer. GNU gzip carries its own deflate and MiGz uses the JVM's zlib, so those two stay system provided.

Running

python3 tools/fetch_migz.py
python3 tools/build_tools.py
curl -LO https://mirror.circlestorm.org/silesia.tar
./bench.py silesia.tar -o results/all-tools.json
python3 scripts/graph_runs.py results/all-tools.json

Any files passed are concatenated into one stream. With no input files the benchmark instead generates a deterministic synthetic mix of source-like text, build-log lines, and random blocks, weighted to land near the 4.8 to 1 of the gzip-ng README corpus under gzip -6.

--levels, --threads, --runs, --size-mb, and --blocks-mb trim or grow the matrix, ./bench.py --help lists the defaults. Everything needs only the Python standard library.

Comparing

python3 scripts/compare_runs.py before.json after.json

Rows match by benchmark name, so two runs of the same matrix, two machines, or two gzip-ng builds, compare directly with time and ratio deltas.

Graphing

scripts/graph_runs.py turns a run into a speed versus ratio SVG, the level ladders connected in order, a decompression throughput panel including the cross-decode pairs, thread scaling line panels, deflate block census panels, block counts and average block sizes across the normal, rsyncable, and independent modes, repetition error bars, and machine specs. An aggregate table prints to stdout.

Results

All tools on the Silesia corpus (silesia.tar, 202 MiB), the common ground of compression benchmarks, built and run as above:

All tools, compress and decompress

About

Whole-process gzip CLI benchmarks across implementations

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages