#hash #fast #cdc #chunking #gear

gearhash

Fast, SIMD-accelerated hash function for content-defined chunking

3 releases

✓ Uses Rust 2018 edition

0.1.2 Dec 13, 2019
0.1.1 Dec 13, 2019
0.1.0 Dec 9, 2019

#189 in Cryptography

MIT/Apache

23KB
601 lines

gearhash

The GEAR hashing function is a fast, rolling hash function that is well suited for content defined chunking.

In particular, this function is used as a building block for the FastCDC algorithm.

The implementation provided in this crate consists of both a simple, scalar variant, as well as optimized versions for the SSE4.2 and AVX2 instruction sets.

Usage

use gearhash::Hasher;

// set up initial state
let mut chunks = vec![];
let mut offset = 0;

// create new hasher
let mut hasher = Hasher::new();

// loop through all matches, and push the corresponding chunks
while let Some(boundary) = hasher.next_match(&buf[offset..], MASK) {
    chunks.push(&buf[offset..offset + boundary]);
    offset += boundary;
}

// push final chunk
chunks.push(&buf[offset..]);

License

This project is licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Dependencies