#compression #lz4 #compress #decompression #decompress

lz4_flex

Fastest LZ4 implementation in Rust, no unsafe by default

16 releases (4 breaking)

0.7.0 Dec 27, 2020
0.6.1 Dec 22, 2020
0.5.1 Dec 16, 2020
0.4.3 Dec 12, 2020
0.3.6 Oct 19, 2020

#27 in Compression

Download history 99/week @ 2020-10-16 18/week @ 2020-10-23 29/week @ 2020-10-30 19/week @ 2020-11-06 1/week @ 2020-11-13 2/week @ 2020-11-20 13/week @ 2020-11-27 59/week @ 2020-12-04 144/week @ 2020-12-11 345/week @ 2020-12-18 273/week @ 2020-12-25 207/week @ 2021-01-01 373/week @ 2021-01-08

271 downloads per month

MIT license

66KB
1K SLoC

Rust Docs Crates.io

lz4_flex

lz4_flex_logo

Fastest LZ4 implementation in Rust. Originally based on redox-os' lz4 compression, but now a complete rewrite. The results in the table are from a benchmark in this project (66Kb JSON).

Compressor Compression Decompression
lz4_flex unsafe 711 MiB/s 3733 MiB/s
lz4_flex safe 628 MiB/s 1433 MiB/s
lz4_cpp 914 MiB/s 3793 MiB/s
lz4_fear 443 MiB/s 836 MiB/s

Features

  • Very good logo
  • LZ4 Block format
  • High performance
  • 0,5s clean release build time
  • Feature flags to configure safe/unsafe code usage
  • no-std support (thanks @coolreader18)
  • 32-bit support

Usage:

Compression and decompression uses no usafe via the default feature flags "safe-encode" and "safe-decode". If you need more performance you can disable them (e.g. with no-default-features).

Safe:

lz4_flex = { version = "0.7.0" }

Performance:

lz4_flex = { version = "0.7.0", default-features = false }
use lz4_flex::{compress_prepend_size, decompress_size_prepended};

fn main(){
    let input: &[u8] = b"Hello people, what's up?";
    let compressed = compress_prepend_size(input);
    let uncompressed = decompress_size_prepended(&compressed).unwrap();
    assert_eq!(input, uncompressed);
}

Benchmarks

The benchmark is run with criterion, the test files are in the benches folder.

Currently 3 implementations are compared, this one, the redox-version, lz-fear and the c++ version via rust bindings. The lz4-flex version is tested with the feature flags safe-decode and safe-encode switched on and off.

Results v0.4.1 10-12-2020 (safe-decode and safe-encode off)

cargo bench --no-default-features

Executed on Core i7-6700 Win10 WSL.

Compress

Decompress

Results v0.4 04-12-2020 (safe-decode and safe-encode on)

cargo bench

Executed on Core i7-6700 Win10 WSL.

Compress

Decompress

Miri

Miri can be used to find issues related to incorrect unsafe usage:

MIRIFLAGS="-Zmiri-disable-isolation -Zmiri-disable-stacked-borrows" cargo miri test --no-default-features

Fuzzer

This fuzz target fuzzes, and asserts compression and decompression returns the original input. cargo fuzz run fuzz_roundtrip

This fuzz target fuzzes, and asserts compression with cpp and decompression returns the original input. cargo fuzz run fuzz_roundtrip_cpp_compress

TODO

  • Frame format
  • High compression
  • Dictionary Compression

No runtime deps