2 releases
0.1.1 | Feb 7, 2024 |
---|---|
0.1.0 | Jan 30, 2024 |
#358 in Build Utils
19KB
279 lines
protobuf-zmq-rust-generator
This crate works with prost
to develop a service generator for
a ZeroMQ + Protobuf implementation, aiding in efficient data
transmission between processes via sockets. It supports both the pub-sub and request-reply patterns.
Originally designed to facilitate communication between a NodeJS client and a Rust server, this package can be adapted to any language that adheres to this protocol.
How to Use
- Install this crate as a
build-dependency
in yourCargo.toml
file. - Create a
build.rs
file in your project's root. Look to prost-build documentation for detailed instructions. - Utilize our service generator during the build process:
prost_build::Config::new() // Optional: defaults to $OUT, can be changed for autocomplete support .out_dir(out_dir) .service_generator(Box::new(ZmqServerGenerator {})) // here .compile_protos(& ["your_proto_file.proto"], & ["your/proto/location/"])
- The generator will create a
your_proto_file.rs
in theout_dir
, containing the generated code.
Now to operate a service server:
- Import the
[Method]Handler
trait and implement it for responses. - For data publication, use the resulting server's
publish
method for eachpubsub
method in your.proto
file.
Check our test files for comprehensive examples.
Implementation Details
In this section, we will discuss the design decisions that went into this package. It's not necessary to understand every detail to use this package, but it may be helpful to understand its limitations.
Requirements
- Aim: Enable inter-process communication with minimal modifications when extending the API
- Provide type safety
- Support creation of a stream for data across different processes subscribed to a specific topic.
- Enable easy creation of asynchronous request-reply tasks across processes.
Given these, we have 2 patterns in operation:
1. Pub/Sub Pattern
- A PUBLISHER application binds to a socket. Any number of SUBSCRIBER applications can connect.
- For communication, the ZMQ frame protocol should be:
[methodName, Output]
, in bytesmessage EmptyInput {} message SubscriptionItem { string data = 1; } service MyServerService { rpc SubscribeToItems(EmptyInput) returns (stream SubscriptionItem) {} }
The data transferred should be ["SubscribeToItems", SubscriptionItem]
.
- Pub-sub methods should start with "SubscribeTo...". Later we will provide a idiomatic way to define this leveraging protobuf options.
- Clients can subscribe and filter events using the
methodName
message. - The
.proto
file defined return type should be a data stream.
2. Request/Reply Pattern
- ROUTER/DEALER sockets are used to allow asynchronous requests.
- A server should handle multiple requests concurrently.
- The ZMQ frame protocol should be:
[requestId, BLANK, methodName, Input]
, in bytes. The server should reply with[clientId, requestId, Output]
The transferred data for this example should bemessage MyRequestInput { int32 time_to_sleep = 1; } message MyRequestResult { bool all_ok = 1; string message = 2; } service MyServerService { rpc MyRequestMethod(MyRequestInput) returns (MyRequestResult) {} }
[requestId, BLANK, "MyRequestMethod", MyRequestInput]
.requestId
is a randomly generated string by the clientBLANK
is an empty frame, used to mimic the original protocol for REQUEST/REPLY patterns.clientId
is included by default by clients. ROUTER should also include this in the reply to ensure the correct dispatching of the reply to a client.
It's possible to use both patterns on the same service: Note: Currently, we only support building Server implementations with this package. Future updates may include client implementations.
Resources
- protobuf-zmq-ts-transport: The NodeJS implementation that permits us to communicate using this protocol
- ZeroMQ: The messaging library used to transmit data between processes
- Prost: The library used to generate Rust code from protobuf files
Contributing
We welcome contributions to this project!
Dependencies
~7–17MB
~229K SLoC