6 releases (3 breaking)
0.4.0 | Aug 5, 2024 |
---|---|
0.3.1 | Jul 30, 2024 |
0.2.0 | Jul 29, 2024 |
0.1.1 | Jul 25, 2024 |
#30 in Database implementations
115KB
2K
SLoC
🦁 buffdb 🦁
buffdb is experimental software. Join buffdb’s Discord for help and have a look at things that don’t work yet. Many basic things are not yet decided.
This is an early implementation of a persistence layer for gRPC written in Rust and based on DuckDB. The goal is to abstract a lot of the complexity associated with using protobufs and flattbuffers so that mobile users can go fast.
How to run
To run the server, you need to have Rust installed. Then, with the repository cloned, you can run
cargo run --all-features -- run
This will start the server on [::1]:50051
, storing the key-value pairs in kv_store.db
and
the blob data in blob_store.db
. All three can be configured with command line flags:
--addr
, --kv-store
, and --blob-store
respectively.
To build with optimizations enabled, run cargo build --all-features --release
. The resulting
binary will be located at target/release/buffdb
. It is statically linked, so it can be moved
anywhere on your file system without issue.
Prefer to handle the gRPC server yourself? buffdb
can be used as a library as well!
Example library usage in Rust
Run cargo add buffdb tonic tokio futures
to add the necessary dependencies. Then you can execute
the following code, which is placed in src/main.rs
.
use buffdb::backend::DuckDb;
use buffdb::kv::{Key, KeyValue, Value};
use tonic::{Request, IntoRequest};
use futures::{stream, StreamExt as _};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut client = buffdb::transitive::kv_client::<_, DuckDb>("kv_store.db").await?;
client
.set(stream::iter([KeyValue {
key: "key_set".to_owned(),
value: "value_set".to_owned(),
}]))
.await?
.into_inner();
let mut stream = client
.get(stream::iter([Key {
key: "key_get".to_owned(),
}]))
.await?
.into_inner();
let Value { value } = stream.next().await.unwrap()?;
assert_eq!(value, "value_get");
Ok(())
}
This project is inspired by conversations with Michael Cahill, Professor of Practice, School of Computer Science, University of Sydney
Command line interface
You can use buffdb help
to see the commands and flags permitted. The following operations are
currently supported:
buffdb run [ADDR]
, starting the server. The default address is[::1]:50051
.buffdb kv get <KEY>
, printing the value to stdout.buffdb kv set <KEY> <VALUE>
, setting the value.buffdb kv delete <KEY>
, deleting the value.buffdb kv eq [KEYS]...
, exiting successfully if the values for all provided keys are equal. Exits with an error code if any two values are not equal.buffdb kv not-eq [KEYS]...
, exiting successfully if the values for all provided keys are unique. Exits with an error code if any two values are equal.buffdb blob get <ID>
, printing the data to stdout. Note that this is arbitrary bytes!buffdb blob store <FILE> [METADATA]
, storing the file (use-
for stdin) and printing the ID to stdout. Metadata is optional.buffdb blob update <ID> data <FILE>
, updating the data of the blob. Use-
for stdin. Metadata is unchanged.buffdb blob update <ID> metadata [METADATA]
, updating the metadata of the blob. Data is unchanged. Omitting[METADATA]
will set the metadata to null.buffdb blob update <ID> all <FILE> [METADATA]
, updating both the data and metadata of the blob. For<FILE>
, use-
for stdin. Omitting[METADATA]
will set the metadata to null.buffdb blob delete <ID>
, deleting the blob.buffdb blob eq-data [IDS]...
, exiting successfully if the blobs for all provided IDs are equal. Exits with an error code if any two blobs are not equal.buffdb blob not-eq-data [IDS]...
, exiting successfully if the blobs for all provided IDs are unique. Exits with an error code if any two blobs are equal.
Commands altering a store will exit with an error code if the key/id does not exist. An exception to this is updating the metadata of a blob to be null, as it is not required to exist beforehand.
All commands for kv
and blob
can use -s
/--store
to specify which store to use. The defaults
are kv_store.db
and blob_store.db
respectively. To select a backend, use -b
/--backend
. The
default varies by which backends are enabled.
Background
This project was inspired by our many edge customers of ours dealing with the challenges associated with low-bandwidth and high performance. We hope that we can build a solution that is helpful for teams tageting edge computing environments.
Today, buffdb’s primary focus is speed: we try to ensure some level of durability for which we pay a performance penalty, but our goal is to eventually be faster than any other embedded database.
High-level Goals
- Reducing the overhead of serialization/deserialization.
- Ensuring consistent data formats between local storage and network communication.
- Providing faster read/write operations compared to JSON or XML.
- Compact Data Storage: ProtoBufs can significantly reduce the size of stored data.
- Interoperability: Seamless integration between the app’s local storage and backend systems.
Use Cases
- Offline Data Access: For apps that need to function offline (e.g., note-taking apps, games, fieldwork, airline, collaborative documents, etc.).
- IoT: For managing device configurations and states locally before syncing with cloud servers.
Dependencies
~29–42MB
~559K SLoC