VennDB

An append-only in-memory database in Rust for rows queried using bit (flag) columns. This database is designed for a very specific use case where you have mostly static data that you typically load at startup and have to query constantly using very simple filters. Datasets like these can be large and should be both fast and compact.

For the limited usecases where venndb can be applied to, it has less dependencies and is faster then traditional choices, such as a naive implementation or a more heavy lifted dependency such as Sqlite.

See the benchmarks for more information on this topic.

This project was developed originally in function of rama, where you can see it being used for example to provide an in-memory (upstream) proxy database. Do let us know in case you use it as well in your project, such that we can assemble a showcase list.

💬 Come join us at Discord on the #venndb public channel. To ask questions, discuss ideas and ask how venndb may be useful for you.

Index

venndb manual:

• Usage: quick introduction on how to use venndb;
• Benchmarks: benchmark results to give you a rough idea how venndb peforms for the use case it is made for (write once, read constantly, using binary filters mostly);
• Q&A: Frequently Asked Questions (FAQ);
• Example: the full example (expanded version from Usage), tested and documented;
• Generated Code Summary: a documented overview of the API that venndb will generate for you when using #[derive(VennDB)] on your named field struct;

technical information:

• Safety
• Compatibility
• MSRV (older versions might work but we make no guarantees);
• Roadmap
• License: MIT license and Apache 2.0 License

misc:

• Contributing
• Sponsors

Usage

Add venndb as a dependency:

cargo add venndb

and import the derive macro in the module where you want to use it:

use venndb::VennDB

#[derive(Debug, VennDB)]
pub struct Employee {
    #[venndb(key)]
    id: u32,
    name: String,
    is_manager: Option<bool>,
    is_admin: bool,
    #[venndb(skip)]
    foo: bool,
    #[venndb(filter, any)]
    department: Department,
    #[venndb(filter)]
    country: Option<String>,
}

fn main() {
    let db = EmployeeDB::from_iter(/* .. */);

    let mut query = db.query();
    let employee = query
        .is_admin(true)
        .is_manager(false)
        .department(Department::Engineering)
        .execute()
        .expect("to have found at least one")
        .any();

    println!("non-manager admin engineer: {:?}", employee);
}

See the full example or the "Generated Code Summary" chapter below to learn how to use the VennDB and its generated code.

Benchmarks

Benchmarks displayed here are taken on a dev machine with following specs:

Macbook Pro — 16 inch (2023)
Chip: Apple M2 Pro
Memory: 16 GB
OS: Sonoma 14.2

The benchmarks tests 3 different implementations of a proxy database

• venndb version (very similar to the example below)
• a naive version, which is just a Vec<Proxy>, over which is iterated
• an sqlite version (using the sqlite crate (version: 0.34.0))

The benchmarks are created by:

1. running just bench;
2. copying the output into ./scripts/plot_bench_charts and running it.

Snippet that is ran for each 3 implementations:

fn test_db(db: &impl ProxyDB) {
    let i = next_round();

    let pool = POOLS[i % POOLS.len()];
    let country = COUNTRIES[i % COUNTRIES.len()];

    let result = db.get(i as u64);
    divan::black_box(result);

    let result = db.any_tcp(pool, country);
    divan::black_box(result);

    let result = db.any_socks5_isp(pool, country);
    divan::black_box(result);
}

Benchmark Performance Results

Performance for Database with 100 records:

Performance for Database with 12_500 records:

Performance for Database with 100_000 records:

We are not database nor hardware experts though. Please do open an issue if you think these benchmarks are incorrect or if related improvements can be made. Contributions in the form of Pull requests are welcomed as well.

See the Contribution guidelines for more information.

Benchmark Allocations Results

Allocations for Database with 100 records:

Allocations for Database with 12_500 records:

Allocations for Database with 100_000 records:

We are not database nor hardware experts though. Please do open an issue if you think these benchmarks are incorrect or if related improvements can be made. Contributions in the form of Pull requests are welcomed as well.

See the Contribution guidelines for more information.

Q&A

❓ Why use this over Database X?

venndb is not a database, but is close enough for some specific purposes. It shines for long-lived read-only use cases where you need to filter on plenty of binary properties and get a rando matching result.

Do not try to replace your usual database needs with it.

❓ Where can I propose a new feature X or some other improvement?

Please open an issue and also read the Contribution guidelines. We look forward to hear from you.

Alternatively you can also join our Discord and start a conversation / discussion over there.

❓ Can I use whatever type for a #[venndb(filter)] property?

Yes, as long as it implements PartialEq + Eq + Hash + Clone. That said, we do recommend that you use enum values if you can, or some other highly restricted form.

Using for example a String directly is a bad idea as that would mean that bE != Be != BE != Belgium != Belgique != België. Even though these are really referring all to the same country. In such cases a much better idea is to at the very least create a wrapper type such as struct Country(String), to allow you to enforce sanitization/validation when creating the value and ensuring the hashes will be the same for those values that are conceptually the same.

❓ How do I make a filter optional?

Both filters (bool properties) and filter maps (T != bool properties with the #[venndb(filter)] attribute) can be made optional by wrapping the types with Option, resulting in Option<bool> and Option<T>.

Rows that have the Option::None value for such an optional column cannot filter on that property, but there is no other consequence beyond that.

❓ Why can do keys have to be unique and non-optional?

Within venndb keys are meant to be able to look up, a row which was previously received via filters.

As such it makes no sense for such keys to be:

• duplicate: it would mean: as that can result in multiple rows or the wrong row to be returned;
• optional: as that would mean the row cannot be looked up when the key is not defined;

❓ How can I allow some rows to match for any value of a certain (filter) column?

Filter maps can allow to have a value to match all other values. It is up to you to declare the filter as such, and to also define for that type what the one value to rule them all is.

Usage:

use venndb::{Any, VennDB};
#[derive(Debug, PartialEq, Eq, Clone, Hash)]
pub enum Department {
  Any,
  Hr,
  Engineering,
}

impl Any for Department {
  fn is_any(&self) -> bool {
    self == Department::Any
  }
}

#[derive(Debug, VennDB)]
pub struct Employee {
  name: String,
  #[venndb(filter, any)]
  department: Department,
}

let db = EmployeeDB::from_iter([
  Employee { name: "Jack".to_owned(), department: Department::Any },
  Employee { name: "Derby".to_owned(), department: Department::Hr },
]);
let mut query = db.query();

// will match Jack and Derby, as Jack is marked as Any, meaning it can work for w/e value
let hr_employees: Vec<_> = query.department(Department::Hr).execute().unwrap().iter().collect();
assert_eq!(hr_employees.len(), 2);

❓ How can I provide custom validation of rows prior to them getting appended?

Is is possible to validate a row based on one or multiple of its properties? Validate in function of relationship between multiple properties? Is it possible to provide custom validation to prevent rows from getting appended that do not adhere to custom validation rules?

Yes to all of the above.

Example:

#[derive(Debug, VennDB)]
#[venndb(validator = my_validator_fn)]
pub struct Value {
   pub foo: String,
   pub bar: u32,
}

fn my_validator_fn(value: &Value) -> bool {
    !value.foo.is_empty() && value.bar > 0
}

let mut db = ValueDB::default();
assert!(db.append(Value {
    foo: "".to_owned(),
    bar: 42,
}).is_err()); // fails because foo == empty

❓ Why do any filter values only match rows that have an any value for that property?

Let's say I have the following struct:

use venndb::{Any, VennDB};

#[derive(Debug, VennDB)]
pub struct Value {
   #[venndb(filter, any)]
   pub foo: MyString,
   pub bar: u32,
}

#[derive(Debug)]
pub struct MyString(String);

impl Any for MyString {
    fn is_any(&self) -> bool {
        self.0 == "*"
    }
}

let db = ValueDB::from_iter([
    Value {
        foo: MyString("foo".to_owned()),
        bar: 8,
    },
    Value {
        foo: MyString("*".to_owned()),
        bar: 16,
    }
].into_Iter()).unwrap();

let mut query = db.query();
query.foo(MyString("*".to_owned()));
let value = query.execute().unwrap().any();
// this will never match the row with bar == 8,
// tiven foo != an any value
assert_eq!(value.bar, 16);

Why is this true? Because it is correct.

Allowing it also to match the value foo would unfairly give more chances for foo to be selected over the any value. This might not seem like a big difference, but it is. Because what if we generate a random string for Values with an _any value? If we would allow all rows to be matched then that logic is now rigged, with a value of foo being more likely then other strings.

As such the only correct answer when filtering for any value, is to return rows that have any value.

❓ How can I query on multiple variants of a "filter map" property?

Just call the query method multiple times. It will allow you to match rows that have either of these values.

Example

use venndb::{Any, VennDB};

#[derive(Debug, VennDB)]
pub struct Value {
   #[venndb(filter)]
   pub foo: String,
   pub bar: u32,
}

let db = ValueDB::from_iter([
    Value {
        foo: "a".to_owned(),
        bar: 8,
    },
    Value {
        foo: "b".to_owned(),
        bar: 12,
    },
    Value {
        foo: "c".to_owned(),
        bar: 16,
    },
].into_Iter()).unwrap();

let mut query = db.query();
query.foo(MyString("a".to_owned()));
query.foo(MyString("c".to_owned()));
let values: Vec<_> = query.execute().unwrap().iter().collect();
assert_eq!(values.len(), 2);
assert_eq!(values[0].bar, 8);
assert_eq!(values[0].bar, 16);

Example

Here follows an example demonstrating all the features of VennDB.

If you prefer a summary of what is generated, or do not understand something from the example below, you can also read the "Generated Code Summary" chapter below.

use itertools::Itertools;
use venndb::VennDB;

#[derive(Debug, VennDB)]
// These attributes are optional,
// e.g. by default the database would be called `EmployeeDB` (name + 'DB').
#[venndb(name = "EmployeeInMemDB", validator = employee_validator)]
pub struct Employee {
    // you can use the `key` arg to be able to get an `Employee` instance
    // directly by this key. It will effectively establishing a mapping from key to a reference
    // of that Employee in the database. As such keys have to have unique values,
    // or else you get an error while appending / creating the DB.
    //
    // NOTE: keys do not only have to be unique, they also have to implement `Clone`!!
    //
    // A property cannot be a filter and a key at the same time,
    // trying to do so will result in a compile-team failure.
    #[venndb(key)]
    id: u32,
    name: String,
    is_manager: bool,
    is_admin: bool,
    // filter (booleans) can be made optional,
    // meaning that the row will not be able to be filtered (found)
    // on this column when the row has a `None` value for it
    is_active: Option<bool>,
    // booleans are automatically turned into (query) filters,
    // use the `skip` arg to stop this. As such it is only really needed for
    // bool properties :)
    #[venndb(skip)]
    foo: bool,
    // non-bool values can also be turned into filters, turning them into 2D filters.
    // For each uniquely inserted Department variant that is inserted,
    // a new filter is kept track of. This allows you to apply a (query) filter
    // based on department, a pretty useful thing to be able to do.
    //
    // NOTE: this does mean that such filter-map types have to also be:
    // `PartialEq + Eq + Hash + Clone`!!
    //
    // A property cannot be a filter and a key at the same time,
    // trying to do so will result in a compile-team failure.
    #[venndb(filter)]
    department: Department,
    // similar to regular bool filters,
    // filter maps can also be optional.
    // When a filter map is optional and the row's property for that filter is None,
    // it will not be registered and thus not be able to filtered (found) on that property
    #[venndb(filter)]
    country: Option<String>,
}

fn employee_validator(employee: &Employee) -> bool {
    employee.id > 0
}

fn main() {
    let db = EmployeeInMemDB::from_iter([
        RawCsvRow("1,John Doe,true,false,true,false,Engineering,USA"),
        RawCsvRow("2,Jane Doe,false,true,true,true,Sales,"),
        RawCsvRow("3,John Smith,false,false,,false,Marketing,"),
        RawCsvRow("4,Jane Smith,true,true,false,true,HR,"),
        RawCsvRow("5,John Johnson,true,true,true,true,Engineering,"),
        RawCsvRow("6,Jane Johnson,false,false,,false,Sales,BE"),
        RawCsvRow("7,John Brown,true,false,true,false,Marketing,BE"),
        RawCsvRow("8,Jane Brown,false,true,true,true,HR,BR"),
    ])
    .expect("MemDB created without errors (e.g. no duplicate keys)");

    println!(">>> Printing all employees...");
    let all_employees: Vec<_> = db.iter().collect();
    assert_eq!(all_employees.len(), 8);
    println!("All employees: {:#?}", all_employees);

    println!(">>> You can lookup an employee by any registered key...");
    let employee = db
        .get_by_id(&2)
        .expect("to have found an employee with ID 2");
    assert_eq!(employee.name, "Jane Doe");

    println!(">>> Querying for all managers...");
    let mut query = db.query();
    query.is_manager(true);
    let managers: Vec<_> = query
        .execute()
        .expect("to have found at least one")
        .iter()
        .collect();
    assert_eq!(managers.len(), 4);
    assert_eq!(
        managers.iter().map(|e| e.id).sorted().collect::<Vec<_>>(),
        [1, 4, 5, 7]
    );

    println!(">>> Querying for all managers with a last name of 'Johnson'...");
    let managers_result = query
        .execute()
        .expect("to have found at least one")
        .filter(|e| e.name.ends_with("Johnson"))
        .expect("to have found a manager with a last name of Johnson");
    let managers = managers_result.iter().collect::<Vec<_>>();
    assert_eq!(managers.len(), 1);
    assert_eq!(managers.iter().map(|e| e.id).collect::<Vec<_>>(), [5]);

    println!(">>> You can also just get the first result if that is all you care about...");
    let manager = managers_result.first();
    assert_eq!(manager.id, 5);

    println!(">>> Querying for a random active manager in the Engineering department...");
    let manager = query
        .reset()
        .is_active(true)
        .is_manager(true)
        .department(Department::Engineering)
        .execute()
        .expect("to have found at least one")
        .any();
    assert!(manager.id == 1 || manager.id == 5);

    println!(">>> Optional bool filters have three possible values, where None != false. An important distinction to make...");
    let mut query = db.query();
    query.is_active(false);
    let inactive_employees: Vec<_> = query
        .execute()
        .expect("to have found at least one")
        .iter()
        .collect();
    assert_eq!(inactive_employees.len(), 1);
    assert_eq!(inactive_employees[0].id, 4);

    println!(">>> If you want you can also get the Employees back as a Vec, dropping the DB data all together...");
    let employees = db.into_rows();
    assert_eq!(employees.len(), 8);
    assert!(employees[1].foo);
    println!("All employees: {:?}", employees);

    println!(">>> You can also get the DB back from the Vec, if you want start to query again...");
    // of course better to just keep it as a DB to begin with, but let's pretend this is ok in this example
    let mut db = EmployeeInMemDB::from_rows(employees).expect("DB created without errors");
    assert_eq!(db.iter().count(), 8);

    println!(">>> Querying for all active employees in the Sales department...");
    let mut query = db.query();
    query.is_active(true);
    query.department(Department::Sales);
    let sales_employees: Vec<_> = query
        .execute()
        .expect("to have found at least one")
        .iter()
        .collect();
    assert_eq!(sales_employees.len(), 1);
    assert_eq!(sales_employees[0].name, "Jane Doe");

    println!(">>> Filter maps that are optional work as well, e.g. you can query for all employees from USA...");
    query.reset().country("USA".to_owned());
    let usa_employees: Vec<_> = query
        .execute()
        .expect("to have found at least one")
        .iter()
        .collect();
    assert_eq!(usa_employees.len(), 1);
    assert_eq!(usa_employees[0].id, 1);

    println!(">>> At any time you can also append new employees to the DB...");
    assert_eq!(EmployeeInMemDBErrorKind::DuplicateKey, db
        .append(RawCsvRow("8,John Doe,true,false,true,false,Engineering,"))
        .unwrap_err().kind());
    println!(">>> This will fail however if a property is not correct (e.g. ID (key) is not unique in this case), let's try this again...");
    assert!(db
        .append(RawCsvRow("9,John Doe,false,true,true,false,Engineering,"))
        .is_ok());
    assert_eq!(db.len(), 9);

    println!(">>> Rows are also validated prior to appending in case a validator is defined...");
    println!("    The next insertion will fail due to the id being zero, a condition defined in the custom validator...");
    assert_eq!(EmployeeInMemDBErrorKind::InvalidRow, db
        .append(RawCsvRow("0,John Doe,true,false,true,false,Engineering,"))
        .unwrap_err().kind());

    println!(">>> This new employee can now also be queried for...");
    let mut query = db.query();
    query.department(Department::Engineering).is_manager(false);
    let new_employee: Vec<_> = query
        .execute()
        .expect("to have found at least one")
        .iter()
        .collect();
    assert_eq!(new_employee.len(), 1);
    assert_eq!(new_employee[0].id, 9);

    println!(">>> You can also extend it using an IntoIterator...");
    db.extend([
        RawCsvRow("10,Glenn Doe,false,true,true,true,Engineering,"),
        RawCsvRow("11,Peter Miss,true,true,true,true,HR,USA"),
    ])
    .unwrap();
    let mut query = db.query();
    query
        .department(Department::HR)
        .is_manager(true)
        .is_active(true)
        .is_admin(true);
    let employees: Vec<_> = query
        .execute()
        .expect("to have found at least one")
        .iter()
        .collect();
    assert_eq!(employees.len(), 1);
    assert_eq!(employees[0].id, 11);

    println!(">>> There are now 2 employees from USA...");
    query.reset().country("USA".to_owned());
    let employees: Vec<_> = query
        .execute()
        .expect("to have found at least one")
        .iter()
        .collect();
    assert_eq!(employees.len(), 2);
    assert_eq!(
        employees.iter().map(|e| e.id).sorted().collect::<Vec<_>>(),
        [1, 11]
    );

    println!(">>> All previously data is still there as well of course...");
    query
        .reset()
        .is_active(true)
        .is_manager(true)
        .department(Department::Engineering);
    let managers: Vec<_> = query
        .execute()
        .expect("to have found at least one")
        .iter()
        .collect();
    assert_eq!(managers.len(), 2);
    assert_eq!(
        managers.iter().map(|e| e.id).sorted().collect::<Vec<_>>(),
        [1, 5]
    );
}

#[derive(Debug)]
struct RawCsvRow<S>(S);

impl<S> From<RawCsvRow<S>> for Employee
where
    S: AsRef<str>,
{
    fn from(RawCsvRow(s): RawCsvRow<S>) -> Employee {
        let mut parts = s.as_ref().split(',');
        let id = parts.next().unwrap().parse().unwrap();
        let name = parts.next().unwrap().to_string();
        let is_manager = parts.next().unwrap().parse().unwrap();
        let is_admin = parts.next().unwrap().parse().unwrap();
        let is_active = match parts.next().unwrap() {
            "" => None,
            s => Some(s.parse().unwrap()),
        };
        let foo = parts.next().unwrap().parse().unwrap();
        let department = parts.next().unwrap().parse().unwrap();
        let country = match parts.next().unwrap() {
            "" => None,
            s => Some(s.to_string()),
        };
        Employee {
            id,
            name,
            is_manager,
            is_admin,
            is_active,
            foo,
            department,
            country,
        }
    }
}

#[derive(Debug, PartialEq, Eq, Hash, Clone)]
pub enum Department {
    Engineering,
    Sales,
    Marketing,
    HR,
}

impl std::str::FromStr for Department {
    type Err = ();

    fn from_str(s: &str) -> Result<Self, Self::Err> {
        match s {
            "Engineering" => Ok(Department::Engineering),
            "Sales" => Ok(Department::Sales),
            "Marketing" => Ok(Department::Marketing),
            "HR" => Ok(Department::HR),
            _ => Err(()),
        }
    }
}

Generated Code Summary

In this chapter we'll list the API as generated by VennDB for the following example code from above:

#[derive(Debug, VennDB)]
#[venndb(name = "EmployeeInMemDB", validator = employee_validator)]
pub struct Employee {
    #[venndb(key)]
    id: u32,
    name: String,
    is_manager: bool,
    is_admin: bool,
    is_active: Option<bool>,
    #[venndb(skip)]
    foo: bool,
    #[venndb(filter)]
    department: Department,
    country: Option<String>,
}

The following public-API datastructures will be generated:

• struct EmployeeInMemDB: the database, that can be used to query (by filters) or look up data (by keys);
• enum EmployeeInMemDBError: the error type that is returned when mutating the DB and a property of the to be inserted row;
• enum EmployeeInMemDBErrorKind: the kind of error that can happen as described for EmployeeInMemDBError;
• struct EmployeeInMemDBQuery: the query builder that is used to build a query that can be executed to query data from the db using filters;
• struct EmployeeInMemDBQueryResult: the result when querying using EmployeeInMemDBQuery and at least one row was found that matched the defined filters;
• struct EmployeeInMemDBQueryResultIter: the iterator type that is used when calling EmployeeInMemDBQueryResult::iter. It has no methods/api other then the fact that it is an Iterator and can be used as one;

The visual specifiers of these datastructures will be the same as the struct that the VennDB macro is applied to. E.g. in this example Employee has a specifier of pub so the above datastructures and their public-apy methods will also be pub.

There are also some other helper datastructures generated — all prefixed with the database name, e.g. EmployeeInMemDB in this example — but we do not mention here as they should not be relied upon and given the prefix it should cause no conflict. In case you do not want to expose these structures to the outside you can wrap your struct within its own mod (module).

Generated Code Summary: Method API

Database: (e.g. EmployeeInMemDB):

Query (e.g. EmployeeInMemDBQuery)

Query Result (e.g. EmployeeInMemDBQueryResult)

⛨ | Safety

This crate uses #![forbid(unsafe_code)] to ensure everything is implemented in 100% safe Rust.

🦀 | Compatibility

venndb is developed mostly on MacOS M-Series machines and run in production on a variety of Linux systems. Windows support is not officially guaranteed, but is tested using Github Actions with success.

Please open a ticket in case you have compatibility issues for your setup/platform. Our goal is not to support all possible platformns in the world, but we do want to support as many as we reasonably can.

Minimum supported Rust version

venndb's MSRV is 1.75.

Using GitHub Actions we also test if venndb on that version still works on the stable and beta versions of rust as well.

🧭 | Roadmap

Please refer to https://github.com/plabayo/venndb/milestones to know what's on the roadmap. Is there something not on the roadmap for the next version that you would really like? Please create a feature request to request it and become a sponsor if you can.

💼 | License

This project is dual-licensed under both the MIT license and Apache 2.0 License.

👋 | Contributing

🎈 Thanks for your help improving the project! We are so happy to have you! We have a contributing guide to help you get involved in the venndb project.

Contributions often come from people who already know what they want, be it a fix for a bug they encountered, or a feature that they are missing. Please do always make a ticket if one doesn't exist already.

It's possible however that you do not yet know what specifically to contribute, and yet want to help out. For that we thank you. You can take a look at the open issues, and in particular:

• good first issue: issues that are good for those new to the venndb codebase;
• easy: issues that are seen as easy;
• mentor available: issues for which we offer mentorship;
• low prio: low prio issues that have no immediate pressure to be finished quick, great in case you want to help out but can only do with limited time to spare;

In general, any issue not assigned already is free to be picked up by anyone else. Please do communicate in the ticket if you are planning to pick it up, as to avoid multiple people trying to solve the same one.

Should you want to contribure this project but you do not yet know how to program in Rust, you could start learning Rust with as goal to contribute as soon as possible to venndb by using "the Rust 101 Learning Guide" as your study companion. Glen can also be hired as a mentor or teacher to give you paid 1-on-1 lessons and other similar consultancy services. You can find his contact details at https://www.glendc.com/.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in venndb by you, shall be licensed as both MIT and Apache 2.0, without any additional terms or conditions.

Acknowledgements

Special thanks goes to all involved in developing, maintaining and supporting the Rust programming language. Also a big shoutout to the "Write Powerful Rust Macros" book by Sam Van Overmeire, which gave the courage to develop this crate.

Some code was also copied/forked from google/argh, for which thank you, we are big fans of that crate. Go use it if you want to create a CLI App.

💖 | Sponsors

venndb is completely free, open-source software which needs lots of effort and time to develop and maintain.

Support this project by becoming a sponsor. One time payments are accepted at GitHub as well as at "Buy me a Coffee".

Sponsors help us continue to maintain and improve venndb, as well as other Free and Open Source (FOSS) technology. It also helps us to create educational content such as https://github.com/plabayo/learn-rust-101, and other open source frameworks such as https://github.com/plabayo/rama.

Sponsors receive perks and depending on your regular contribution it also allows you to rely on us for support and consulting.

Finally, you can also support us by shopping Plabayo <3 VennDB merchandise 🛍️ at https://plabayo.threadless.com/.