#low-latency #data-flow #distributed #zenoh #operator #data-pipeline #composable

dora-coordinator

dora goal is to be a low latency, composable, and distributed data flow

47 releases

new 0.3.9 Feb 6, 2025
0.3.8 Dec 11, 2024
0.3.7 Nov 18, 2024
0.3.5 Jul 3, 2024
0.1.0 Jun 24, 2022

#256 in Operating systems

Download history 21/week @ 2024-11-04 103/week @ 2024-11-11 168/week @ 2024-11-18 23/week @ 2024-11-25 16/week @ 2024-12-02 271/week @ 2024-12-09 24/week @ 2024-12-16 4/week @ 2025-01-06 4/week @ 2025-01-13 251/week @ 2025-02-03

255 downloads per month
Used in dora-cli

Apache-2.0

140KB
3.5K SLoC

Coordinator

Prototype for a process/library-based dora-rs implementation, instead of framework-based. The idea is that each operator is compiled as a separate executable. The dora-coordinator runtime is responsible for reading the dataflow descriptor file and launching the operators accordingly. The operators use a common library called dora-api, which implements the communication layer based on zenoh.

This approach has the following advantages:

  • Less overhead
    • No data transfer between a runtime and the operator
    • The compiler can inline and optimize the full process
  • More flexibility
    • Operators can be sync or async
    • They can decide how many threads and which execution model they use
    • The OS ensures fair share of resources (e.g. CPU time) -> no need to cooperate with other operators
    • Operators get all inputs immediately -> no need for input rules
    • Keeping local state is easily possible
  • Separate address spaces
    • The operators are isolated from each other.

There are drawbacks too, for example:

  • Less control
    • Processes run independently -> need to cooperate with the runtime, e.g. on stop signals
    • Operator migration is more difficult
  • Operators are always isolated
    • No way of using in-memory channels
    • Local sockets and shared memory should be still possible

Dependencies

~13–44MB
~697K SLoC