#memory #data #reduce #stack #transparently #deque

compressed_collections

Collections which transparently compress data to reduce memory usage

1 unstable release

0.4.0 Jan 4, 2023

#468 in Compression

MIT/Apache

19KB
360 lines

Collections which transparently compress data to reduce memory usage

This create offers collections which automatically compress themselves, which reduces memory usage, sometimes substantially, allowing collections to be held in memory that would otherwise be too big.

The only restriction on the datatypes in the collections is that they must be serde serializable.

For instance:

use compressed_collections::Stack;

let mut compressed_stack = Stack::new();
for _ in 0..(1024 * 1024 * 100) {
    compressed_stack.push(1);
}

This only allocates around 10MB (the default buffer size), whereas the equivalent vector would be around 100MB in size.

Design goals:

  • Provide collections with a subset of the API of the standard equivalent wherever possible for easy dropin use.
  • Only implement the efficient operations for each datastructure

Datastructures:

  • Stack
  • Deque
  • Map

License

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Dependencies

~6.5MB
~167K SLoC