3 releases
0.1.2 | Aug 31, 2023 |
---|---|
0.1.1 | Aug 27, 2023 |
0.1.0 | Aug 27, 2023 |
#948 in Database interfaces
27KB
244 lines
Bicycle 🚲
The high level goals of this project are to create a database which is, simple and fast. In order to achieve these goals it will be built in Rust, atop RocksDB and exist as a gRPC server.
For more information and a better writeup, look here: https://problemchild.engineering/2023-08-19-the-database
For some really imprecise (just run on my local with ghz), initial benchmarks you can also look here: https://hachyderm.io/@seanwatters/110948629015247923
Usage
Usage is super clunky rn bc this lib only has one consumer (me). But if you're interested in trying it, run the following:
cargo run --package bicycle_cli -- create cli/test.proto
That will create a server binary and proto file for your consuming services. So in the cli/out/
you'll have server
and bicycle.proto
.
The bicycle.proto
is what any developer who is familiar with gRPC can use to code-gen and build a client to the database. Right now, the database
is very light weight and has no administration infrastructure, permissions or auth; I get away with this because I'm only ever running it in private
subnets within the same VPCs on AWS and stuff. But there is always room for evolution.
Running
Once RocksDB is finally done building (holy fuck that takes way too long and I need to figure out how to like cache it or some do some other Rustacean magic to make it stop), you should be able to run the server with:
./out/server
Clients
./out/bicycle.proto
Because the database server is just a gRPC server, you can use all native gRPC libraries for any language you like.
and you can also roll over to your preferred gRPC GUI client, type in localhost::50051
, AND because we implement
server reflection, when you plug in the URL it will automatically load up all your available RPCs (assuming your client GUI supports that).
Example
from blog post
The simplest example here (because right now it's "just a kv store" and I'm not gonna explain STD to everyone rn) is just 1 model,
so we're gonna pick Dog
.
Schema
// cli/test.proto
syntax = "proto3";
package bicycle;
message Dog {
string pk = 1;
string name = 2;
uint32 age = 3;
string breed = 4;
}
If we run our above commands to generate the ./out/server
and ./out/bicycle.proto
we can get the actually useful proto definition for our client (./out/bicycle.proto
):
cargo run --package bicycle_cli -- create cli/test.proto
Proto
When we run our script with the schema.proto
, it barfs out this new proto (bicycle.proto
). In this new proto, are the actual primitives for database
interaction and has all the fun stuff we need to do in application-land.
This file shouldn't ever need to be modified by the developer directly and could break a lotta stuff.
syntax = "proto3";
package bicycle;
message Dogs {
repeated Dog dogs = 1;
}
message Dog {
string pk = 1;
string name = 2;
uint32 age = 3;
string breed = 4;
}
message IndexQuery {
oneof expression {
string eq = 1;
string gte = 2;
string lte = 3;
string begins_with = 4;
}
}
message Empty {}
service Bicycle {
rpc GetDogsByPk(IndexQuery) returns (Dogs) {}
rpc DeleteDogsByPk(IndexQuery) returns (Empty) {}
rpc PutDog(Dog) returns (Empty) {}
rpc BatchPutDogs(Dogs) returns (Empty) {}
}
Client
I was gonna put together a full on code example for every language but that's exhausting and grpcurl
gives you the idea.
Basically we have 4 RPCs for each model:
GetXByPk
DeleteXByPk
PutX
BatchPutX
And then you have the IndexQuery
helper which basically allows you to do key-range queries.
Here are the really basic examples:
## PutDog
grpcurl -plaintext -d '{
"pk": "DOG#1",
"name": "Rover",
"age": 3,
"breed": "Golden Retriever"
}' localhost:50051 bicycle.Bicycle.PutDog
## BatchPutDogs
grpcurl -plaintext -d '{
"dogs": [
{
"pk": "DOG#2",
"name": "Buddy",
"age": 2,
"breed": "Labrador"
},
{
"pk": "DOG#3",
"name": "Max",
"age": 4,
"breed": "Poodle"
}
]
}' localhost:50051 bicycle.Bicycle.BatchPutDogs
## GetDogs
grpcurl -plaintext -d '{"begins_with": "DOG#"}' localhost:50051 bicycle.Bicycle.GetDogsByPk
## DeleteDogs
grpcurl -plaintext -d '{"eq": "DOG#3"}' localhost:50051 bicycle.Bicycle.DeleteDogsByPk
Contributing
there isn't really a thing right now. it's kinda just one person rn with a few others maybe joining. we gotta set up a CLA or something i think. but check back in.
Dependencies
~38–51MB
~1M SLoC