#compression #lz4 #compress #decompression #decompress

lz4_flex

Fastest LZ4 implementation in Rust, no unsafe by default

23 releases (5 breaking)

new 0.8.2 Jun 9, 2021
0.8.1 May 30, 2021
0.7.5 Jan 21, 2021
0.7.0 Dec 27, 2020
0.3.6 Oct 19, 2020

#15 in Compression

Download history 410/week @ 2021-02-24 353/week @ 2021-03-03 329/week @ 2021-03-10 327/week @ 2021-03-17 383/week @ 2021-03-24 464/week @ 2021-03-31 397/week @ 2021-04-07 714/week @ 2021-04-14 847/week @ 2021-04-21 707/week @ 2021-04-28 722/week @ 2021-05-05 698/week @ 2021-05-12 574/week @ 2021-05-19 426/week @ 2021-05-26 1014/week @ 2021-06-02 809/week @ 2021-06-09

2,561 downloads per month
Used in 7 crates (3 directly)

MIT license

145KB
2.5K SLoC

Rust Docs Crates.io

lz4_flex

lz4_flex_logo

Fastest LZ4 implementation in Rust. Originally based on redox-os' lz4 compression, but now a complete rewrite. The results in the table are from a benchmark in this project (66Kb JSON) on with the block format.

Compressor Compression Decompression Ratio
lz4_flex unsafe 898 MiB/s 4417 MiB/s 0.2289
lz4_flex unsafe w. checked_decode 898 MiB/s 4023 MiB/s 0.2289
lz4_flex safe 724 MiB/s 2014 MiB/s 0.2289
lzzz (lz4 1.9.3) 993 MiB/s 4193 MiB/s 0.2283
lz4_fear 456 MiB/s 976 MiB/s 0.2283
snap 861 MiB/s 941 MiB/s 0.2242

Features

  • Very good logo
  • LZ4 Block format
  • LZ4 Frame format (thanks @arthurprs)
  • High performance
  • 1,5s clean release build time
  • Feature flags to configure safe/unsafe code usage
  • no-std support (thanks @coolreader18)
  • 32-bit support

Usage:

Compression and decompression uses no usafe via the default feature flags "safe-encode" and "safe-decode". If you need more performance you can disable them (e.g. with no-default-features).

Safe:

lz4_flex = { version = "0.8.0" }

Performance:

lz4_flex = { version = "0.8.0", default-features = false }

Warning: If you don't trust your input and your are using the Block format, use checked-decode in order to avoid out of bounds access. When using the Frame format make sure to enable checksums.

lz4_flex = { version = "0.8.0", default-features = false, features = ["checked-decode"] }
use lz4_flex::{compress_prepend_size, decompress_size_prepended};

fn main(){
    let input: &[u8] = b"Hello people, what's up?";
    let compressed = compress_prepend_size(input);
    let uncompressed = decompress_size_prepended(&compressed).unwrap();
    assert_eq!(input, uncompressed);
}

Benchmarks

The benchmark is run with criterion, the test files are in the benches folder.

Currently 4 implementations are compared, this one, lz-fear and the c++ version via rust bindings and snappy. The lz4-flex version is tested with the feature flags safe-decode and safe-encode switched on and off.

Results v0.8.0 17-05-2021 (safe-decode and safe-encode off)

cargo bench --no-default-features

Executed on Core i7-6700 Linux Mint.

Compress

Decompress

Results v0.8.0 17-05-2021 (safe-decode and safe-encode on)

cargo bench

Executed on Core i7-6700 Linux Mint.

Compress

Decompress

Miri

Miri can be used to find issues related to incorrect unsafe usage:

MIRIFLAGS="-Zmiri-disable-isolation -Zmiri-disable-stacked-borrows" cargo miri test --no-default-features

Fuzzer

This fuzz target generates corrupted data for the decompressor. Make sure to switch to the checked_decode version in fuzz/Cargo.toml before testing this. cargo fuzz run fuzz_decomp_corrupt_block and cargo fuzz run fuzz_decomp_corrupt_frame

This fuzz target asserts that a compression and decompression rountrip returns the original input. cargo fuzz run fuzz_roundtrip and cargo fuzz run fuzz_roundtrip_frame

This fuzz target asserts compression with cpp and decompression with lz4_flex returns the original input. cargo fuzz run fuzz_roundtrip_cpp_compress

TODO

  • High compression

Dependencies

~150KB