6 releases (breaking)

0.7.0 Nov 21, 2023
0.6.0 Nov 20, 2023
0.5.0 Oct 2, 2023
0.4.0 Sep 5, 2023
0.1.0 Jul 20, 2023

#273 in Compression

Download history 18/week @ 2023-08-16 9/week @ 2023-08-23 18/week @ 2023-08-30 23/week @ 2023-09-06 37/week @ 2023-09-13 11/week @ 2023-09-20 35/week @ 2023-09-27 22/week @ 2023-10-04 11/week @ 2023-10-11 20/week @ 2023-10-18 14/week @ 2023-10-25 15/week @ 2023-11-01 17/week @ 2023-11-08 63/week @ 2023-11-15 32/week @ 2023-11-22 28/week @ 2023-11-29

140 downloads per month
Used in 2 crates (via bustools)

GPL-3.0-or-later

40KB
451 lines

NewPFD-rs

Rust library implementing the NewPFD integer compression/decompression algorithm.

Performance

It's currently lacking optimization for speed, but it's decently fast. We perform this on geometrically distributed integers (Geo(lambda=0.01)) to force encoding exceptions in the NewPFD-block.

  • Encoding: 90ms/ 1M integers
  • Decoding: 16ms/ 1M integers

See benchmarks for details.

Examples

For more examples, see the rust-docs.

// Encode some data using NewPFD
use newpfd::newpfd_bitvec::{encode, decode};
let data = vec![10_u64,12,10,1,1,2,3];
let blocksize = 32; // needs to be a mutliple of 32

// encode
let (compressed_data, _) = encode(data.iter().cloned(), blocksize);
// compressed_data is a `bitvec::BitVec` (similar to a Vec<bool>)

// decode
let (decompressed_data, bits_processed) = decode(&compressed_data, data.len(), blocksize);
assert_eq!(data, decompressed_data);
assert_eq!(compressed_data.len(), bits_processed); // the entire bitstream was consumed

Dependencies

~2.5MB
~48K SLoC