2 releases
0.1.1 | Dec 10, 2023 |
---|---|
0.1.0 | Dec 7, 2023 |
#29 in #reusable
58KB
951 lines
Sisyphus Tasks
Utilities for long-running, resilient tasks.
This library contains code I wrote, found useful, and want to keep using. It aims to provide systems
The general idiom is focused on spawning long-lived worker loops, which use channels to communicate.
Understanding this crate
High Level Example
pub struct MyWorker{
pipe: Pipe<WorkItem>,
}
impl Boulder for MyWorker {
fn spawn(self) -> JoinHandle<Fall<Self>> {
tokio::spawn(async move {
// pipe gives refs to items
while let Some(item) = self.pipe.next().await {
// worker does work on each item as it becomes available
self.do_work_on(item);
// do some async work too :)
self.async_work_as_well(item).await;
}
});
}
}
let task = MyWorker::new(a_pipe).run_forever();
Current Utils:
-
Sisyphus
- A scaffolding system for spawning long-lived, recoverable tasks
- A
Boulder
is a looping, fallible task Boulder
logic is defined in aBoulder::spawn()
method that returns aJoinHandle
- A
Fall
is an error in that task.Fall::Recoverable
- errors that the task believes it can recoverFall::Unrecoverable
- errors that the task believes it cannot recover
- The
Boulder::run_until_panic
method handles restarting on recoverableFall
s, and reporting unrecoverable errors Boulder
s may define custom recovery or cleanup logicBoulder
s may also panic. In that case, noFall
is generated, and the- panic is propagated upward
Sisyphus
manages aBoulder
loop. He exposes an interface to observe its status and abort the work
-
Pipe
- An inbound and an outbound channel
- Enforce process-once semantics
- Prevents data loss on worker error
- Designed for relatively linear data-processing pipelines
- e.g. retrieval -> metrics -> indexing -> other handling
- Convenience methods for running synchronous and asynchronous
for_each
on channel contents
Future Utils:
- Abstraction layers for instantiating complex pipes from lists of Sisyphuses
- Pipes should allow sync & async transforms (inbound
T
, outboundU
)
Copyright Note
Some code descends from utilities written for Nomad. It is used and reproduced under its license terms.
Dependencies
~11–22MB
~307K SLoC