block-db

Local multi-threaded byte DB.

Description

Operates by providing a directory for the BlockDB structure, and can then read_bytes, write_bytes, delete_bytes, and batch based on DataBlock IDs; which are in the format: {DataFile ID}:{UUIDv4}. To see all the methods provided:

• Full crate documentation can be found here at docs.rs

Installation

cargo add block-db

Overview

• Each BlockDB has many DataFiles, and their pointers are WALed for persistence.

• Each Chunk of a DataBlock is 4096 bytes.

• Each DataFile is a maximum of 5gb, but will overflow for the last block, or the first block if above 5gb.

• When a DataBlock is deleted, it becomes a free block(NULL) that can then be filled in subsequent writes.

Usage Example

// mut access is only needed for `batch` operations
let mut block_db = BlockDB::open("./my_db").await?;

let data_block_id_1: String = block_db.write_bytes(b"BlockDB").await?;

// Some(vec![66, 108, 111, 99, 107, 68, 66])
let bytes: Option<Vec<u8>> = block_db.read_bytes(&data_block_id_1).await?;

block_db.delete_bytes(&data_block_id_1).await?;

let data_block_id_2: String = block_db.write_bytes(b"Hello World").await?;

let new_data_block_ids: Vec<String> = block_db.batch(
    vec![ // Writes
        b"Some data",
        b"Some more data",
    ],
    vec![ // Deletes
        &data_block_id_2,
    ]
).await?;

Roadmap

• Patches

• Optimizations

Contributing

Open to any contributions. All tests must pass, and the new features or changes should "make sense" based on the current API.

Note: The tests::smoke_test::smoke_test will take quite awhile to run as it will create a in-memory 5gb vector to write to and read from a DataFile; so it is recommended to use cargo test --release.

License

MIT License

Copyright (c) 2024 Robert Lopez

See LICENSE.md

Project status

I plan to continue maintaining this project for the foreseeable future.