LFCS 21 🐧 Modern Compression — zstd, lz4
The compression landscape has shifted. For two decades, gzip and bzip2 were the standard tools, and xz was the heavy hitter for maximum ratio. But the modern workloads demand something different: compression that is fast enough for real-time data, for container images, for log streams, and for the kernel’s own initramfs. The two algorithms that have filled this gap are zstd (Zstandard) and lz4. Both were created by Yann Collet, both are open source, and both prioritize speed alongside ratio. They are the compression tools that the LFCS exam expects you to know in addition to the older standards.
lz4 is the speed champion. It compresses at over 500 MB/s per core and decompresses at multiple GB/s, which means the decompression is often fast enough to be I/O bound rather than CPU bound . The ratio is modest — roughly 2.1 on the Silesia corpus — but the speed is the point. lz4 is used where the data must be compressed and decompressed constantly: the filesystem snapshots, the database pages, the network buffers, and the kernel’s own initramfs in some distributions . zstd is the ratio champion. It compresses at 510 MB/s at level 1, with a ratio of 2.9, and its decompression is 1550 MB/s . It offers a very wide range of compression levels, from the ultra-fast negative levels (--fast=1, --fast=4) to the maximum ratio levels (--ultra, up to 22), and its decompression speed is preserved across all the levels .
This chapter covers three areas. First, why the modern compressors exist — the speed-versus-ratio trade-off and the workloads that need each point on the curve. Second, how to use zstd and lz4 — the basic compression and decompression, the level flags, the streaming, and the integration with tar. Third, the patterns and the edge cases — the file naming conventions, the dictionary compression for small data, the rsyncable mode, and the choices between the tools. The chapter ends with a complete example session, a quick reference, best practices, common pitfalls, real-world examples, and diagrams showing the speed and ratio comparison.
Key point: zstd and lz4 are the modern compression tools, both created by Yann Collet. lz4 prioritizes extreme speed (>500 MB/s compression, multiple GB/s decompression) with a modest ratio (~2.1). zstd prioritizes a better balance (510 MB/s at level 1, ratio 2.9) and offers a wide range of levels. Both integrate with tar via the --zstd and --lz4 options. The default file extensions are .zst and .lz4.
Why the modern compressors exist
The speed problem. gzip compresses at around 105 MB/s and decompresses at 390 MB/s on a modern CPU . That is fast enough for a nightly backup, but it is not fast enough for the workloads that compress data constantly. A database that compresses every page it writes to disk cannot afford to spend 10 milliseconds per page on compression. A log shipper that compresses every line cannot afford to be the bottleneck. A container runtime that decompresses an image layer on every container start cannot afford a slow decompression. The modern workloads demand compression that is an order of magnitude faster, and lz4 and zstd provide it.
The ratio problem. The speed comes at a cost. lz4 compresses to a ratio of 2.1, while gzip compresses to 2.7 and xz to a much higher ratio. The data that is compressed with lz4 takes more space. For the workloads where the space matters — the long-term archives, the cold storage, the distribution packages — the ratio is the priority, and xz or zstd --ultra is the right choice. For the workloads where the speed matters — the hot data, the frequent access, the real-time streams — the ratio is secondary, and lz4 is the right choice. The two tools exist because the two workloads exist.
The decompression problem. lz4‘s decompression is the fastest of any general-purpose compressor, reaching 3850 MB/s on the Silesia corpus . The decompression is often faster than the disk read, which means the compression is effectively free: the data is read from the disk, decompressed, and the decompression is not the bottleneck. zstd‘s decompression is 1550 MB/s at level 1, which is also faster than most disks. The decompression speed is preserved across the compression levels for both tools, so the level can be raised for a better ratio without slowing the decompression .
The kernel and initramfs problem. The Linux kernel’s initramfs is compressed with zstd in many distributions, including Ubuntu and Fedora. The benchmarking showed that zstd produces a much smaller initramfs than lz4, and while the decompression is slower, the loading time dominates: the kernel spends 80-90% of the boot time loading the image into memory, and the decompression is only 10-20% . The smaller image is faster to load, and the loading time wins. The lz4 is used where the decompression must be instantaneous, and zstd is used where the size matters more.
The tar problem. The tar command integrates both tools with the --zstd and --lz4 options, and the file extensions are .tar.zst and .tar.lz4. The integration is the same as the -z for gzip and the -j for bzip2. The tar command pipes the archive through the compressor, and the decompression is automatic when the file extension is recognized .
The trade-off. zstd and lz4 are not replacements for gzip and xz. They are additions. The gzip is still the universal format, understood by every system. The xz is still the best ratio for the long-term archives. The zstd and lz4 are the tools for the modern workloads where the speed matters, and the choice between them is the choice between the fastest (lz4) and the better balance (zstd).
a. Using lz4
lz4 compresses a single file by default, and the original file is preserved. The compressed file has the .lz4 extension .
lz4 file.txtThis creates file.txt.lz4 and keeps file.txt. The -z flag forces compression when the input is already a .lz4 file, and the --rm flag erases the source file on completion.
The -d flag decompresses, and the unlz4 command is equivalent:
lz4 -d file.txt.lz4
unlz4 file.txt.lz4The -m flag enables multiple files, which compresses each file into its own .lz4 file:
lz4 -m file1.txt file2.txtThe -m -d decompresses multiple .lz4 files. The -m --rm gives a behavior closer to gzip: multiple files, and the source files are erased .
The -1 to -12 flags set the compression level, and the -9 and -12 use the high-compression (HC) mode:
lz4 -9 file.txt
lz4 -12 file.txtThe -t flag tests the integrity of a .lz4 file without decompressing it. The lz4cat command decompresses to standard output, and the -c flag writes to standard output without creating a file.
b. Using zstd
zstd compresses a file and creates the .zst extension. The original file is removed by default, and the -k flag keeps it.
zstd file.txt # creates file.txt.zst, removes file.txt
zstd -k file.txt # creates file.txt.zst, keeps file.txtThe -d flag decompresses, and the unzstd command is equivalent:
zstd -d file.txt.zst
unzstd file.txt.zstThe -1 to -19 flags set the compression level, and the --ultra flag enables the levels 20 to 22:
zstd -1 file.txt # fastest
zstd -19 file.txt # high ratio
zstd --ultra -22 file.txt # maximum ratioThe negative levels, --fast=1 to --fast=4, are faster than level 1 and offer a lower ratio . The -T# flag sets the number of threads, and the -T0 uses the number of physical CPU cores. The -t flag tests the integrity, and the -l flag lists the information about a .zst file.
The zstd command supports the streaming mode, where the input is from a pipe and the output is to a pipe:
tar -cf - directory/ | zstd -o backup.tar.zstThe -o flag specifies the output file. The --rsyncable flag makes the compressed file more friendly to rsync, at a negligible cost to the compression ratio .
The dictionary compression is the zstd feature for the small data. The --train flag trains a dictionary from a set of samples, and the -D flag uses the dictionary:
zstd --train samples/* -o dictionary
zstd -D dictionary file.txtThe dictionary improves the ratio dramatically for the small files that share the structure, such as the JSON records or the log lines .
c. The integration with tar and the choices
The tar command integrates both tools. The --zstd option pipes the archive through zstd, and the --lz4 option pipes it through lz4.
tar -cf backup.tar --zstd directory/
tar -cf backup.tar --lz4 directory/The -a flag uses the file extension to determine the compression, so tar -caf backup.tar.zst directory/ uses zstd automatically. The equivalent in the older syntax is the -I flag:
tar -cf backup.tar -I 'zstd -T0' directory/
tar -cf backup.tar -I 'lz4 -9' directory/The -I flag specifies the compressor command, and the tar pipes the archive through it. The decompression is automatic for the standard extensions, and the -I flag is used for the custom options.
The choice between lz4 and zstd depends on the workload. The lz4 is the right choice when the speed is the priority: the real-time streams, the hot data, the frequent access, the decompression that must be instant. The zstd is the right choice when the ratio matters more than the absolute fastest speed: the backups, the distribution packages, the kernel images, the data that is compressed once and decompressed many times.
The choice between zstd and xz depends on the ratio and the speed. The xz produces a smaller file than zstd --ultra, but the compression is much slower, and the decompression is slower too. The zstd is the better choice when the compression time matters, and the xz is the better choice when the file size is the only priority.
Complete Example Session
# ============================================
# PART 1: COMPRESS WITH LZ4
# ============================================
lz4 file.txt
# Creates file.txt.lz4, keeps file.txt
lz4 --rm file.txt
# Creates file.txt.lz4, removes file.txt
lz4 -9 file.txt
# High compression mode
# ============================================
# PART 2: DECOMPRESS WITH LZ4
# ============================================
lz4 -d file.txt.lz4
# Restores file.txt
unlz4 file.txt.lz4
# Equivalent
lz4cat file.txt.lz4
# Decompresses to stdout
# ============================================
# PART 3: COMPRESS WITH ZSTD
# ============================================
zstd file.txt
# Creates file.txt.zst, removes file.txt
zstd -k file.txt
# Creates file.txt.zst, keeps file.txt
zstd -1 file.txt
# Fastest level
zstd -19 file.txt
# High ratio
zstd --ultra -22 file.txt
# Maximum ratio
# ============================================
# PART 4: DECOMPRESS WITH ZSTD
# ============================================
zstd -d file.txt.zst
# Restores file.txt
unzstd file.txt.zst
# Equivalent
zstdcat file.txt.zst
# Decompresses to stdout
# ============================================
# PART 5: MULTITHREADING
# ============================================
zstd -T0 -19 largefile.tar
# Uses all physical cores
lz4 -m -9 file1.txt file2.txt
# Compresses multiple files
# ============================================
# PART 6: THE TAR INTEGRATION
# ============================================
tar -caf backup.tar.zst directory/
# Uses zstd (auto-detected from .zst)
tar -caf backup.tar.lz4 directory/
# Uses lz4 (auto-detected from .lz4)
tar -cf backup.tar -I 'zstd -T0 -19' directory/
# Custom zstd options
tar -xf backup.tar.zst
# Automatic decompression
# ============================================
# PART 7: THE STREAMING
# ============================================
tar -cf - directory/ | zstd -o backup.tar.zst
# Streams the tar archive through zstd
tar -cf - directory/ | lz4 - backup.tar.lz4
# Streams the tar archive through lz4
# ============================================
# PART 8: THE DICTIONARY
# ============================================
zstd --train samples/*.json -o dictionary
# Trains a dictionary from the samples
zstd -D dictionary -19 file.json
# Uses the dictionary for compression
# ============================================
# PART 9: THE RSYNCABLE MODE
# ============================================
zstd --rsyncable -19 backup.tar
# The compressed file is more rsync-friendly
# ============================================
# PART 10: THE COMPARISON
# ============================================
# The speed and ratio comparison (Silesia corpus):
#
# lz4 1.10.0 | Ratio 2.101 | Comp 675 MB/s | Decomp 3850 MB/s
# zstd 1.5.7 -1 | Ratio 2.896 | Comp 510 MB/s | Decomp 1550 MB/s
# zstd --fast=4 | Ratio 2.146 | Comp 665 MB/s | Decomp 2050 MB/s
# gzip 1.3.1 -1 | Ratio 2.743 | Comp 105 MB/s | Decomp 390 MB/s
# xz 5.6.0 | Ratio 3.5+ | Comp 5-20 MB/s | Decomp 400-800 MB/s
#
# lz4: fastest, modest ratio
# zstd: better balance, wide range of levels
# gzip: universal, slower
# xz: best ratio, slowestThe ten parts show the lz4 compression, the lz4 decompression, the zstd compression, the zstd decompression, the multithreading, the tar integration, the streaming, the dictionary, the rsyncable mode, and the comparison.
Quick Reference
lz4 Commands
| Command | Purpose |
|---|---|
lz4 file | Compress to file.lz4 |
lz4 -d file.lz4 | Decompress |
unlz4 file.lz4 | Decompress (alias) |
lz4cat file.lz4 | Decompress to stdout |
lz4 -m file1 file2 | Compress multiple files |
lz4 -9 file | High compression |
lz4 -t file.lz4 | Test integrity |
lz4 --rm file | Remove source on completion |
zstd Commands
| Command | Purpose |
|---|---|
zstd file | Compress to file.zst |
zstd -d file.zst | Decompress |
unzstd file.zst | Decompress (alias) |
zstdcat file.zst | Decompress to stdout |
zstd -19 file | High ratio |
zstd --ultra -22 file | Maximum ratio |
zstd -T0 file | Multithreaded |
zstd -t file.zst | Test integrity |
zstd -k file | Keep source |
The tar Integration
| Command | Effect |
|---|---|
tar -caf backup.tar.zst dir/ | Auto-detect zstd |
tar -caf backup.tar.lz4 dir/ | Auto-detect lz4 |
tar -cf backup.tar -I 'zstd -T0' dir/ | Custom zstd |
tar -cf backup.tar -I 'lz4 -9' dir/ | Custom lz4 |
The Speed and Ratio
| Tool | Ratio | Compression | Decompression |
|---|---|---|---|
| lz4 | 2.101 | 675 MB/s | 3850 MB/s |
| zstd -1 | 2.896 | 510 MB/s | 1550 MB/s |
| zstd –fast=4 | 2.146 | 665 MB/s | 2050 MB/s |
| gzip -1 | 2.743 | 105 MB/s | 390 MB/s |
| xz | 3.5+ | 5-20 MB/s | 400-800 MB/s |
Best Practices
✅ Do This:
# Use lz4 for the real-time data
lz4 -9 hot-data.bin # ✅
# Use zstd for the better balance
zstd -19 backup.tar # ✅
# Use the multithreading for the large files
zstd -T0 -19 largefile.tar # ✅
# Use the tar integration
tar -caf backup.tar.zst directory/ # ✅
# Use the dictionary for the small data
zstd --train samples/* -o dictionary # ✅
# Use the rsyncable mode for the rsync targets
zstd --rsyncable -19 backup.tar # ✅❌ Don’t Do This:
# Don't use lz4 for the long-term archives
lz4 archive.tar # the ratio is poor # ❌
# Don't use zstd --ultra for the real-time data
zstd --ultra -22 hot-data.bin # too slow for the real-time # ❌
# Don't forget the -T0 for the large files on the multi-core systems
zstd -19 largefile.tar # single-threaded # ❌
# Don't use the wrong extension
tar -caf backup.tar.zstd # should be .zst # ❌
# Don't rely on the implicit output in scripts
lz4 file.txt # use the explicit output # ❌Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| The source file is removed | zstd removes by default | Use -k |
| The source file is kept | lz4 keeps by default | Use --rm |
| The wrong extension | Typo | Use .zst or .lz4 |
| The compression is slow | No multithreading | Use -T0 |
| The decompression is slow | The wrong tool | Use lz4 for the decompression |
| The tar does not compress | The wrong -I syntax | Use the correct command |
| The dictionary is ignored | The -D flag is missing | Add -D dictionary |
Real-World Examples
1. lz4 Compression
lz4 file.txt2. zstd Compression
zstd -19 file.txt3. The tar Integration
tar -caf backup.tar.zst directory/4. The Multithreading
zstd -T0 -19 largefile.tar5. The Streaming
tar -cf - directory/ | zstd -o backup.tar.zst6. The Dictionary
zstd --train samples/* -o dictionary7. The Decompression
zstd -d file.txt.zst8. The Test
zstd -t file.txt.zst9. The Rsyncable
zstd --rsyncable -19 backup.tar10. The lz4 High Compression
lz4 -12 file.txtVisual
The Speed and Ratio
┌──────────────────────────────────────────────────────────────┐
│ THE SPEED AND RATIO │
│ │
│ Ratio (higher is better): │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ xz ████████████████████████████████ 3.5+ │ │
│ │ zstd -1 ████████████████████████ 2.9 │ │
│ │ gzip -1 ██████████████████████ 2.7 │ │
│ │ zstd --fast=4█████████████████ 2.1 │ │
│ │ lz4 █████████████████ 2.1 │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ Compression speed (higher is better): │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ lz4 ████████████████████████████████ 675 │ │
│ │ zstd --fast=4███████████████████████████████ 665 │ │
│ │ zstd -1 ████████████████████████ 510 │ │
│ │ gzip -1 █████ 105 │ │
│ │ xz █ 5-20 │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ Decompression speed (higher is better): │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ lz4 ████████████████████████████████ 3850 │ │
│ │ zstd --fast=4████████████████████████████ 2050 │ │
│ │ zstd -1 ████████████████████████ 1550 │ │
│ │ gzip -1 ██████ 390 │ │
│ │ xz ████████ 400-800│ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ lz4: fastest, modest ratio. │
│ zstd: better balance, wide range of levels. │
│ │
└──────────────────────────────────────────────────────────────┘The zstd Levels
┌──────────────────────────────────────────────────────────────┐
│ THE zstd LEVELS │
│ │
│ --fast=4 ── --fast=1 ── 1 ── 3 ── 9 ── 19 ── --ultra 22 │
│ │ │ │ │ │ │ │ │
│ Fastest Fast Bal Bal High Max Max │
│ Lowest Low Ratio │
│ Ratio Ratio │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Compression speed: │ │
│ │ --fast=4: 665 MB/s │ │
│ │ -1: 510 MB/s │ │
│ │ -19: 5-20 MB/s │ │
│ │ --ultra 22: 1-5 MB/s │ │
│ │ │ │
│ │ Decompression speed is roughly constant │ │
│ │ across all levels: 400-800 MB/s (server), │ │
│ │ 1550 MB/s (desktop, level 1). │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ The compression level trades the speed for the ratio. │
│ The decompression speed is preserved. │
│ │
└──────────────────────────────────────────────────────────────┘The Tar Integration
┌──────────────────────────────────────────────────────────────┐
│ THE TAR INTEGRATION │
│ │
│ tar -caf backup.tar.zst directory/ │
│ │ │
│ │ The -a flag detects the .zst extension. │
│ │ The tar pipes the archive through zstd. │
│ ▼ │
│ tar archive ──► zstd ──► backup.tar.zst │
│ │
│ tar -xf backup.tar.zst │
│ │ │
│ │ The tar detects the .zst extension. │
│ │ The tar pipes the file through zstd -d. │
│ ▼ │
│ backup.tar.zst ──► zstd -d ──► tar archive │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ The same works for .lz4, .gz, .bz2, .xz. │ │
│ │ The -a flag makes the extension the contract. │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────┘The Dictionary Compression
┌──────────────────────────────────────────────────────────────┐
│ THE DICTIONARY COMPRESSION │
│ │
│ The small data is hard to compress. │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ A 1KB JSON record: │ │
│ │ {"name": "...", "email": "...", "age": ...} │ │
│ │ │ │
│ │ The compression algorithm has no "past" to learn │ │
│ │ from. The ratio is poor. │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ The dictionary provides the "past". │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ zstd --train samples/*.json -o dictionary │ │
│ │ → the dictionary learns the common structure │ │
│ │ │ │
│ │ zstd -D dictionary file.json │ │
│ │ → the ratio improves dramatically │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ The more data-specific the dictionary, the better. │
│ There is no universal dictionary. │
│ │
└──────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| lz4 | Fastest compression; modest ratio |
| zstd | Better balance; wide range of levels |
| lz4 extension | .lz4 |
| zstd extension | .zst |
| lz4 decompression | unlz4, lz4cat |
| zstd decompression | unzstd, zstdcat |
| tar integration | --zstd, --lz4, -a, -I |
| Multithreading | -T0 for zstd |
| Dictionary | --train, -D for zstd |
| Rsyncable | --rsyncable for zstd |
Key takeaways:
lz4is the speed champion. It compresses at over 500 MB/s per core and decompresses at multiple GB/s, which is often faster than the disk read. The ratio is modest, but the speed is the point for the real-time data, the hot data, and the frequent access .zstdis the better balance. It compresses at 510 MB/s at level 1 with a ratio of 2.9, and its decompression is 1550 MB/s. It offers a wide range of levels, from the ultra-fast negative levels to the maximum ratio levels, and the decompression speed is preserved across all levels .- Both tools integrate with
tar. The--zstdand--lz4options pipe the archive through the compressor, and the-aflag detects the extension. The file extensions are.tar.zstand.tar.lz4. - The choice between
lz4andzstddepends on the workload. Thelz4is the right choice when the speed is the priority. Thezstdis the right choice when the ratio matters more than the absolute fastest speed. - The dictionary compression improves the small data. The
zstd --traintrains a dictionary from the samples, and the-Dflag uses it. The ratio improves dramatically for the small files that share the structure . - The
--rsyncablemode makes the compressed file more rsync-friendly. The compression state is periodically synchronized, and the impact on the ratio is negligible . - The
lz4andzstdare the modern additions to the compression toolkit. Thegzipis still the universal format, and thexzis still the best ratio for the long-term archives. Thelz4andzstdfill the gap for the workloads where the speed matters. - The LFCS exam expects the knowledge of the modern tools. The “Archive, compress, and extract files” objective covers the tools, and the
tarintegration is the standard pattern .
Remember: lz4 and zstd are the modern compression tools that prioritize speed. The lz4 is the fastest, with a modest ratio. The zstd is the better balance, with a wide range of levels and a preserved decompression speed. Both integrate with tar via the --zstd and --lz4 options, and the file extensions are .tar.zst and .tar.lz4. The dictionary compression improves the small data, and the --rsyncable mode makes the compressed file more rsync-friendly. The choice between the two depends on whether the speed or the ratio is the priority. For the real-time data, the lz4. For the backups and the distribution packages, the zstd.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!