2 releases

0.1.1 May 22, 2023
0.1.0 May 22, 2023

#391 in Compression

Download history 12/week @ 2024-02-22 7/week @ 2024-02-29 1/week @ 2024-03-21 50/week @ 2024-03-28 3/week @ 2024-04-04

53 downloads per month

MIT license

543 lines

Rust Stream VByte encoder/decoder

This library provides encoding/decoding primitives for Stream VByte encoding.

Stream VByte encoding is a SIMD accelerated algorithm of VarInt decompression. It is used in a search and database systems as a way of efficiently store and stream large number of VarInts from a disk or main memory.

Benchmarking results:

CPU Base Freq. (GHz) Turbo Freq. (GHz) Result (GElem/s)
Xeon E3-1245 v5 3.5 3.9 5.0
Core i7-1068NG7 2.3 4.1 5.5

The idea behind VarInt is not to store leading zero bytes of the number. This way large amount of relatively small numbers can be stored in a much more compact way. VarInt encoding is frequently used with delta-encoding if numbers are stored in the ascending order. This way all the numbers are smaller by magnitude, hence better compression.

Stream VByte working using two data streams: control stream and data stream. Control stream contains control words (1 byte each). Each control word describe length of 4 numbers in the data stream (2 bits per number, 00 - length 1, 01 - length 2 and so on).

No runtime deps