#async #io #bytes #concurrent

streamcatcher

A thread-safe, shared (asynchronous), almost-lockless stream buffer

4 releases (1 stable)

1.0.0 Aug 31, 2021
0.1.1 Aug 31, 2021
0.1.0 Sep 27, 2020
0.0.1 Aug 17, 2020

#74 in Concurrency

Download history 342/week @ 2021-07-01 192/week @ 2021-07-08 197/week @ 2021-07-15 202/week @ 2021-07-22 152/week @ 2021-07-29 242/week @ 2021-08-05 190/week @ 2021-08-12 358/week @ 2021-08-19 369/week @ 2021-08-26 247/week @ 2021-09-02 408/week @ 2021-09-09 686/week @ 2021-09-16 462/week @ 2021-09-23 344/week @ 2021-09-30 576/week @ 2021-10-07 593/week @ 2021-10-14

1,213 downloads per month
Used in songbird

MIT/Apache

54KB
1K SLoC

docs-badge crates.io version crates.io downloads license build badge

Streamcatcher

A Rust thread-safe, shared (asynchronous) stream buffer designed to lock only on accessing and storing new data.

Streamcatcher is designed to allow seeking on otherwise one-way streams (e.g., command output) whose output needs to be accessed by many threads without constant reallocations, contention over safe read-only data, or unnecessary stalling. Only threads who read in new data ever need to lock the data structure, and do not prevent earlier reads from occurring.

Features

  • Lockless access to pre-read data and finished streams.
  • Transparent caching of newly read data.
  • Allows seeking on read-only bytestreams.
  • Piecewise allocation to reduce copying and support unknown input lengths.
  • Optional acceleration of reads on stream completion by copying to a single backing store.
  • (Stateful) bytestream transformations.
  • Async support with the "async" feature, and runtimes via ["async-std-compat", "smol-compat", "tokio-compat"].

The main algorithm is outlined in this blog post, with rope reference tracking moved to occur only in the core.

License

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Detailed guidelines are given in the CONTRIBUTING file.

Dependencies

~0.7–2.4MB
~42K SLoC