Transformation Pipeline

Middleware-esque API for transforming data.

Features

• Define a custom data type passed between stages
• Simple pipeline flow exposed to stages: Abort, Finish (early), Skip
• Wrapper functions to run the entire pipeline

Potential Future Features

• Sub-pipelines (e.g. Routers)

Usage

Installation

Add transformation-pipeline to your Cargo.toml file, and add it as an external crate.

extern crate transformation_pipeline;

Define Data

Identify or define a datatype that will be passed between stages.

This can be a built-in, such as String, or a custom struct. However, the datatype must have the Clone trait defined.

use std::clone::Clone;

struct User {
    name: String,
}

impl Clone for User {
    fn clone(&self) -> Self {
        User {
            name: self.name.clone(),
        }
    }
}

Creating Stages

Create structs for each stage of the pipeline, and implement the PipelineStage trait for each stage.

use transformation_pipeline::PipelineStage;
use transformation_pipeline::StageResult;
use transformation_pipeline::StageActions;

struct MyPipelineStage {}

impl TransformationStage<User> for MyPipelineStage {
    fn run(&self, previous: User) -> StageResult<User> {
        let mut name: String = "John ".to_owned();
        name.push_str(&previous.name);
        Ok(StageActions::Next(User {
            name: name,
        }))
    }
}

See stage actions for more information about the different actions that can be returned.

Create a Pipeline

Now you can assemble a pipeline out of the created stages. Each stage must be put in a Box, which is a built-in type.

use transformation_pipeline::TransformationPipeline;
let pipeline: TransformationPipeline<User> = TransformationPipeline::new(vec![
    Box::new(MyPipelineStage {}),
    // Add other stages here:
    // Box::new(MyOtherPipelineStage {})
]);

Using the Pipeline

Now you can pass data into the pipeline:

let input: User = User {
  name: "Doe"
};
let output = pipeline.run(input).unwrap();

assert_eq!(output.name, "John Doe");

Documentation

Stage Actions

Each stage of the pipeline must complete with some "Action".

Next Action

The standard action is "Next", which passes the given data to the next pipeline stage. If the stage is the final stage in the pipeline, the given data is returned as the pipeline result.

Ok(StageActions::Next( /* data for next stage */ ))

Skip Action

A stage can complete with "Skip", which starts the next pipeline stage as if the current stage never existed.

This is equivalent to calling:

return Ok(StageActions::Next(previous));

But it can be a little more explicit to what is happening:

if /* action is already completed */ {
  return Ok(StageActions::Skip);
}
/* Do action */
Ok(StageActions::Next( /* ... */ ))

Finish Action

A stage can cause the pipeline to immediately complete with the "Finish" action. This returns the given data as the pipeline result, and does not run any further stages.

Ok(StageActions::Finish(/* final data */ ))

Jump Action

A stage can skip subsequent steps in the pipeline with the "Jump" action. This passes the given data to a stage further down the pipeline, and doesn't run any stages in between.

// SAME AS Next():
return Ok(StageActions::Skip(0, /* next data */ ));

// Skips 1 stage:
return Ok(StageActions::Skip(1, /* data to pass to the 2nd stage */ ));

Abort Action

A stage can complete with the "Abort" action, causing the entire pipeline to abort with an error.

Ok(StageActions::Abort)

(Anti-)Purpose/Alternatives

This package is not designed to:

• Handle different data types between stages (e.g. successive maps)
• Have multiple functions exposed by pipeline stages (e.g. fancy plugins)

cargo-plugin may be a better alternative for general-purpose plugins.