#dora #distributed

dora-coordinator

dora goal is to be a low latency, composable, and distributed data flow

31 releases

0.3.2 Jan 29, 2024
0.3.0 Nov 3, 2023
0.2.4 Jul 21, 2023
0.2.2-rc Mar 31, 2023
0.1.0 Jun 24, 2022

#2 in #low

Download history 127/week @ 2023-10-29 26/week @ 2023-11-05 30/week @ 2023-11-12 20/week @ 2023-11-19 86/week @ 2023-11-26 28/week @ 2023-12-03 27/week @ 2023-12-10 4/week @ 2023-12-17 52/week @ 2023-12-24 2/week @ 2023-12-31 106/week @ 2024-01-07 8/week @ 2024-01-14 9/week @ 2024-01-21 160/week @ 2024-01-28 36/week @ 2024-02-04 121/week @ 2024-02-11

326 downloads per month
Used in dora-cli

Apache-2.0

54KB
1K SLoC

Coordinator

Prototype for a process/library-based dora-rs implementation, instead of framework-based. The idea is that each operator is compiled as a separate executable. The dora-coordinator runtime is responsible for reading the dataflow descriptor file and launching the operators accordingly. The operators use a common library called dora-api, which implements the communication layer based on zenoh.

This approach has the following advantages:

  • Less overhead
    • No data transfer between a runtime and the operator
    • The compiler can inline and optimize the full process
  • More flexibility
    • Operators can be sync or async
    • They can decide how many threads and which execution model they use
    • The OS ensures fair share of resources (e.g. CPU time) -> no need to cooperate with other operators
    • Operators get all inputs immediately -> no need for input rules
    • Keeping local state is easily possible
  • Separate address spaces
    • The operators are isolated from each other.

There are drawbacks too, for example:

  • Less control
    • Processes run independently -> need to cooperate with the runtime, e.g. on stop signals
    • Operator migration is more difficult
  • Operators are always isolated
    • No way of using in-memory channels
    • Local sockets and shared memory should be still possible

Dependencies

~13–28MB
~410K SLoC