#data-engineering #api-integration #declarative-pipeline

app cargo-pipe

A command-line tool to generate and build data integration app using pipebase framework

17 releases

new 0.1.16 Nov 24, 2021
0.1.15 Sep 24, 2021
0.1.10 Aug 31, 2021

#50 in Cargo plugins

Download history 55/week @ 2021-08-07 42/week @ 2021-08-14 32/week @ 2021-08-21 53/week @ 2021-08-28 41/week @ 2021-09-04 5/week @ 2021-09-11 35/week @ 2021-09-18 25/week @ 2021-09-25 33/week @ 2021-10-02 16/week @ 2021-10-09 1/week @ 2021-10-16 32/week @ 2021-10-23 16/week @ 2021-11-06 12/week @ 2021-11-13 33/week @ 2021-11-20

97 downloads per month

Apache-2.0

175KB
5K SLoC

cargo-pipe is a clap based command-line tool to generate and build pipebase app with manifest

Install Tools

install Rust and Cargo (Minimum Supported Version of Rust 1.54.0)

curl https://sh.rustup.rs -sSf | sh

install rustfmt

rustup component add rustfmt

install cargo-pipe CLI

cargo install cargo-pipe

Usage

cargo pipe --help

Quick Start

go to resources directory

cargo pipe new && \
cargo pipe validate -o -p && \
cargo pipe generate && \
cargo pipe build -o timer -r

run app

./timer

note that, any change to pipe.yml requires re-run validate, generate, build steps

Validation & Debug

validate manifest only

cargo pipe validate -o -p

build with debug flag

cargo pipe build -d
...
     Warning struct is never constructed: `Bar`
     Warning 1 warning emitted

fix the warning by remove objects section

@@ -23,13 +23,6 @@ pipes:
     config:
       ty: PrinterConfig
     upstreams: [ "timer1", "timer2" ]
-objects:
-  - ty: Bar
-    fields:
-      - name: bar1
-        ty: Integer
-      - name: bar2
-        ty: String

since manifest changed, re-generate and build

cargo pipe generate && \
cargo pipe build -d

Describe Manifest

list all pipes and objects

cargo pipe describe -a
...
    Describe pipes
      Result pipe: timer1, printer, timer2
    Describe objects
      Result objects: Bar

describe a pipe

cargo pipe describe -p timer1
...
    Describe pipe timer1
      Result
Name:   timer1
Type:   poller
Config: { type: TimerConfig, path: catalogs/timer1.yml }
Upstream: []

describe an object

cargo pipe describe -o Bar
...
    Describe object Bar
      Result
Type: Bar
Type Metas:
Fields:
Name   Type Metas
bar1    i32
bar2 String

describe pipelines cross a pipe

cargo pipe describe -l printer
...
    Describe pipelines for printer
      Result pipeline: timer2(u128) -> printer
      Result pipeline: timer1(u128) -> printer

Dependencies

~5.5MB
~120K SLoC