Zstandard (zstd) is the latest hot compression algorithm, developed by Yann Collet at Facebook. It can be used in file systems like ZFS and btrfs. But it can also be useful in everyday usage.

In no event, unless required by applicable law or agreed to in writing will I be liable to you for damages, including any general, special, incidental, or consequential damages arising out of the use or inability to use the information, commands, scripts and snippets provided here (including but not limited to loss of data or data being rendered inaccurate, or losses sustained by you or third parties, or a failure of the command/script/snippets to operate with any other programs), even if such holder or other party has been advised of the possibility of such damages.

Writing articles like this one requires time and resources. If you found it helpful or even if you didn't, I'd love to hear from you—whether you have feedback, suggestions, or spotted any bugs or typos. Your input would mean the world to me! You can reach out using the email address listed in the imprint.

Basics

The zstd command is fairly simple to use. Here a little basic use cases:

Compress file

zstd in.tar
# is the same as
zstd in.tar -o in.tar.zst
zstd -z in.tar
zstd --compress in.tar

Decompress file

zstd -d in.tar.zstd
# is the same as
zstd --decompress
zstd --uncompress

Compression level

Ranges from 1 to 19. The default is 3.

zstd -19 in.tar -o out.tar.zstd

But higher compression levels 20-22 can be unlocked (but using a lot more memory). There also exists a "fast" mode (the higher the number after fast the faster the compression speed). And then there is an adaptive mode which adapts the compression level based on perceived I/O conditions. You can give a option min and max value to the automatic adaption.

zstd --ultra -22 in.tar
zstd --fast=1 in.tar
zstd --adapt in.tar
zstd --adapt=3,19 in.tar

--adapt can remain stuck at low speed when used with multiple worker threads

Worker threads

The default number of worker threads is 1. You can adapt them with -T# where # is the number of worker threads. -T0 attempts to detect and use the number of physical CPU cores.

With tar

Built-in support

To tar a file and compress it with zstd (with newer tar versions):

tar --zstd --create --file out.tar.zst <source>

More options

But if we want some more options (like compression level or number of threads) we need to resort to another syntax:

tar -I "zstd -T0 --ultra --21" --create --file out.tar.zstd <source>

To extract we can use the normal tar command: tar --extract --file in.tar.zstd.

Pipes

tar --create <file/folder> | zstd --adapt -o out.tar.zstd
tar --extract --file out.tar.zstd
zstd -d out.tar.zstd --stdout | tar --list
zstd -d out.tar.zstd --stdout | tar --extract

Examples

Copy file from server via clipboard

A little savage example to show some piping

On server:

cat hello.txt | zstd -T0 -19 | base64 -w0 | wl-copy 

On Client:

echo <file content> | base64 -d | zstd -d

# or
echo <file content> | base64 -d | zstd -d
 > hello.txt
# example:
echo KLUv/SQOcQAASGVsbG8sIHdvcmxkIQpX5TSF | base64 -d | zstd -d

Sources

man zstd
man tar