6 releases
0.0.5 | Mar 2, 2024 |
---|---|
0.0.4 | Jan 29, 2024 |
0.0.3 | Aug 15, 2023 |
0.0.2 | Jul 4, 2023 |
0.0.0 | Apr 24, 2023 |
#608 in Rust patterns
773 downloads per month
Used in 3 crates
(via hydroflow)
30KB
1K
SLoC
Hydroflow
Hydroflow is a low-latency dataflow runtime written in Rust. The goal of the Hydro Project is to empower developers to harness the full potential of the cloud by making distributed programs easy to specify and automatic to scale. Hydroflow is the lowest level in the Hydro stack, serving as a single-node low-latency runtime with explicit networking. This allows us to support not just data processing pipelines, but distributed protocols (e.g. Paxos) and real-world long-running applications as well.
Take a look at the Hydroflow Book.
The Hydroflow Surface Syntax
Hydroflow comes with a custom "surface syntax"—a domain-specific language which serves as a very simple, readable IR for specifying single-node Hydroflow programs. These programs are intended to be stitched together by the Hydro stack to create larger autoscaling distributed systems.
Here's a simple example of the surface syntax. Check out the Hydroflow Playground for an interactive demo.
source_iter(0..10)
-> map(|n| n * n)
-> filter(|n| *n > 10)
-> foo;
foo = map(|n| (n..=n+1))
-> flatten()
-> for_each(|n| println!("Howdy {}", n));
For more, check out the surface syntax section of the Hydroflow book.
Start with a Template Program
We provide a cargo-generate
template for you to get started from a simple working example.
To install cargo-generate
, run the following:
cargo install cargo-generate
Then run
cargo generate gh:hydro-project/hydroflow-template
and you will get a well-formed Hydroflow/Rust project to use as a starting point. It provides a simple Echo Server and Client, and advice for adapting it to other uses.
Dev Setup
See the setup section of the book.
The Examples Container
The hydroflow/examples
subdirectory of this repository includes a number of examples.
To make running these examples in the cloud easier, we've created a Docker image that contains compiled versions of those examples. The image is defined in the Dockerfile
in the same directory as this README.
If you want to build the examples container locally, you can run
docker build -t hydroflow-examples .
This will build an image suitable for your architecture.
The scripts/multiplatform-docker-build.sh <image name>
script will build both arm64
and amd64
versions of the image and push them to the image name specified. By default, this will push the image to DockerHub; if you want to push the image to another repository, you can pass an image URL as the argument to multiplatform-docker-build.sh
instead.
Example binaries are located in /usr/src/myapp
.
lib.rs
:
Pusherator generics and argument order conventions:
Next
(being the next owned pusherator) should come first in generic arguments.- However
next: Next
innew(...)
arguments should come last. This is so the rest of the arguments appear in the order data flows in. - Any closures
Func
should come before their arguments, so:<Func: Fn(A) -> B, A, B>
Dependencies
~165KB