16 releases (4 breaking)

0.5.3 Oct 22, 2024
0.5.0 Sep 25, 2024
0.3.1 Jul 30, 2024

#316 in Database interfaces

Download history 282/week @ 2024-07-08 181/week @ 2024-07-15 368/week @ 2024-07-22 452/week @ 2024-07-29 130/week @ 2024-08-05 4/week @ 2024-08-12 142/week @ 2024-09-09 23/week @ 2024-09-16 168/week @ 2024-09-23 232/week @ 2024-09-30 212/week @ 2024-10-07 146/week @ 2024-10-21

594 downloads per month

Apache-2.0

140KB
978 lines

.github/workflows/rust.yml test Crates.io Version Crates.io Downloads (latest version) docs.rs GitHub commit activity Matrix

alt text

valentinus

next generation vector db built with lmdb bindings

dependencies

  • bincode/serde - serialize/deserialize
  • lmdb-rs - database bindings
  • ndarray - numpy equivalent
  • ort/onnx - embeddings

getting started

git clone https://github.com/kn0sys/valentinus && cd valentinus

optional environment variables

var usage default
LMDB_USER working directory of the user for database $USER
LMDB_MAP_SIZE Sets max environment size, i.e. size in memory/disk of all data 20% of available memory
ONNX_PARALLEL_THREADS parallel execution mode for this session 1
VALENTINUS_CUSTOM_DIM embeddings dimensions for custom models all-mini-lm-6 -> 384
VALENTINUS_LMDB_ENV environment for the database (i.e. test, prod) test

tests

  • Note: all tests currently require the all-MiniLM-L12-v2_onnx directory
  • Get the model.onnx and tokenizer.json from huggingface or build them
mkdir all-MiniLM-L12-v2_onnx
cd all-MiniLM-L12-v2_onnx && wget https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2/resolve/main/config.json
wget https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2/resolve/main/onnx/model.onnx
wget https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2/resolve/main/special_tokens_map.json
wget https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2/resolve/main/tokenizer_config.json
wget https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2/resolve/main/tokenizer.json
wget https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2/resolve/main/vocab.txt

RUST_TEST_THREADS=1 cargo test

examples

see examples

reference

inspired by this chromadb python tutorial

Dependencies

~23MB
~468K SLoC