Zstandard

Zstandard , or zstd as short version, is a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios. It's backed by a very fast entropy stage, provided by Huff0 and FSE library. Zstandard's format is stable and documented in RFC8878. Multiple independent implementations are already available. This repository represents the reference implementation, provided as an open-source dual BSD OR GPLv2 licensed C library, and a command line utility producing and decoding .zst , .gz , .xz and .lz4 files. Should your project require another programming language, a list of known ports and bindings is provided on Zstandard homepage. For reference, several fast compression algorithms were tested and compared on a desktop featuring a Core i7-9700K CPU @ 4.9GHz and running Ubuntu 24.04 ( Linux 6.8.0-53-generic ), using [lzbench], an open-source in-memory benchmark by @inikep compiled with [gcc] 14.2.0, on the [Silesia compression corpus]. The negative compression levels, specified with --fast= , offer faster compression and decompression speed at the cost of compression ratio. The following tests were run on a server running Linux Debian ( Linux version 4.14.0-3-amd64 ) with a Core i7-6700K CPU @ 4.0GHz, using [lzbench], an open-source in-memory benchmark by @inikep compiled with [gcc] 7.3.0, on the [Silesia compression corpus]. A few other algorithms can produce higher compression ratios at slower speeds, falling outside of the graph. For a larger picture including slow modes, click on this link. Previous charts provide results applicable to typical file and stream scenarios (several MB). Small data comes with different perspectives. To solve this situation, Zstd offers a training mode , which can be used to tune the algorithm for a selected type of data. Training Zstandard is achieved by providing it with a few samples (one file per sample). The result of this training is stored in a file called "dictionary", which must be loaded before compression and decompression. Using this dictionary, the compression ratio achievable on small data improves dramatically. The following example uses the github-users sample set, created from github public API. It consists of roughly 10K records weighing about 1KB each. These compression gains are achieved while simultaneously providing faster compression and decompression speeds. Training works if there is some correlation in a family of small data samples. The more data-specific a dictionary is, the more efficient it is (there is no universal dictionary ). Hence, deploying one dictionary per type of data will provide the greatest benefits. Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm will gradually use previously decoded content to better compress the rest of the file. make is the main build system of this project. It is the reference, and other build systems are periodically updated to stay compatible. However, small drifts and feature differences can be present, since perfect synchronization is difficult. For this reason, when your build system allows it, prefer employing make . Assuming your system supports standard make (or gmake ), just invoking make in root directory generates zstd cli at root, and also generates libzstd into lib/ . The Makefile follows the GNU Standard Makefile conventions, allowing staged install, standard compilation flags, directory variables and command variables. For advanced use cases, specialized flags which control binary generation and installation paths are documented in lib/README.md for the libzstd library and in programs/README.md for the zstd CLI. A cmake project generator is available for generating Makefiles or other build scripts to create the zstd binary as well as libzstd dynamic and static libraries. The repository root now contains a minimal CMakeLists.txt that forwards to build/cmake , so you can configure the project with a standard cm