Lib.rs

›

#data-integration #tokio #api-integration #declarative-pipeline #data-engineering

pipebase

A tokio based runtime library for data integration app

13 releases

0.2.0	Dec 26, 2021
0.1.11	Dec 4, 2021
0.1.9	Nov 24, 2021
0.1.8	Sep 6, 2021
0.1.5	Aug 22, 2021

#7 in #data-engineering

104 downloads per month
Used in 19 crates

Apache-2.0

180KB
5.5K SLoC

pipebase is a tokio based runtime library for data integration app. It provides basic pipes implemented with rust standard library.

Basic Pipes

list of basic pipes built in pipebase

pipe type	implementation	example
`Exporter`	`Printer`	`printer`
`Poller`	`Timer`	`timer`
`Listener`	`LocalFilePathVisitor`	`file_path`
`Streamer`	`FileLineReader`	`stateless_word_count`
`Streamer`	`IteratorReader`	`stateless_word_count`
`Mapper`	`FieldVisit`	`field_filter`
`Mapper`	`FilterMap`	`field_filter`
`Mapper`	`StringSplitter`	`stateless_word_count`
`Mapper`	`Projection`	`project_file`
`Mapper`	`FileReader`	`project_file`
`Mapper`	`FileWriter`	`convert_csv`
`Mapper`	`Conversion`	`convert_csv`
`Selector`	`RandomSelector`	`ingest_redis_parallel`
`Collector`	`TextCollector`	`ingest_elasticsearch`
`Collector`	`InMemoryWindowCollector`	`moving_average`

Dependencies

~8–15MB
~188K SLoC