2 releases

0.1.1 May 22, 2023
0.1.0 May 22, 2023

#407 in Compression

Download history 37/week @ 2024-03-24 17/week @ 2024-03-31 85/week @ 2024-06-09 4/week @ 2024-06-16

89 downloads per month

MIT license

34KB
543 lines

Rust Stream VByte encoder/decoder

This library provides encoding/decoding primitives for Stream VByte encoding.

Stream VByte encoding is a SIMD accelerated algorithm of VarInt decompression. It is used in a search and database systems as a way of efficiently store and stream large number of VarInts from a disk or main memory.

Benchmarking results:

CPU Base Freq. (GHz) Turbo Freq. (GHz) Result (GElem/s)
Xeon E3-1245 v5 3.5 3.9 5.0
Core i7-1068NG7 2.3 4.1 5.5

The idea behind VarInt is not to store leading zero bytes of the number. This way large amount of relatively small numbers can be stored in a much more compact way. VarInt encoding is frequently used with delta-encoding if numbers are stored in the ascending order. This way all the numbers are smaller by magnitude, hence better compression.

Stream VByte working using two data streams: control stream and data stream. Control stream contains control words (1 byte each). Each control word describe length of 4 numbers in the data stream (2 bits per number, 00 - length 1, 01 - length 2 and so on).

No runtime deps