#tokio #api-integration #declarative-pipeline #data-engineering

pipebase

A tokio based runtime library for data integration app

13 releases

0.2.0 Dec 26, 2021
0.1.11 Dec 4, 2021
0.1.9 Nov 24, 2021
0.1.8 Sep 6, 2021
0.1.5 Aug 22, 2021

#8 in #data-engineering

Download history 119/week @ 2024-04-01 37/week @ 2024-04-08 48/week @ 2024-04-15 47/week @ 2024-04-22 47/week @ 2024-04-29 59/week @ 2024-05-06 49/week @ 2024-05-13 42/week @ 2024-05-20 55/week @ 2024-05-27 50/week @ 2024-06-03 42/week @ 2024-06-10 42/week @ 2024-06-17 63/week @ 2024-06-24 99/week @ 2024-07-08 53/week @ 2024-07-15

219 downloads per month
Used in 19 crates

Apache-2.0

180KB
5.5K SLoC

pipebase is a tokio based runtime library for data integration app. It provides basic pipes implemented with rust standard library.

Basic Pipes

list of basic pipes built in pipebase

pipe type implementation example
Exporter Printer printer
Poller Timer timer
Listener LocalFilePathVisitor file_path
Streamer FileLineReader stateless_word_count
Streamer IteratorReader stateless_word_count
Mapper FieldVisit field_filter
Mapper FilterMap field_filter
Mapper StringSplitter stateless_word_count
Mapper Projection project_file
Mapper FileReader project_file
Mapper FileWriter convert_csv
Mapper Conversion convert_csv
Selector RandomSelector ingest_redis_parallel
Collector TextCollector ingest_elasticsearch
Collector InMemoryWindowCollector moving_average

Dependencies

~7–15MB
~171K SLoC