#data #process #parallel #data-structure

processing_chain

Rust library to set up processing chains of large amounts of data

5 releases

0.2.2 Nov 27, 2022
0.2.1 Nov 20, 2022
0.2.0 Nov 12, 2022
0.1.1 Sep 4, 2022
0.1.0 Sep 4, 2022

#1849 in Data structures

Download history 132/week @ 2024-02-21 39/week @ 2024-02-28

171 downloads per month

MIT license

16KB
324 lines

processing-chain

processing-chain provides a convenient way to seamlessly set up processing chains for large amounts of data.

Please read the API documentation on docs.rs or take a look at the examples.

processing-chain is based on the concept of Item which is an abstraction that is used to spawn all the processes in parallel. All the user needs to do is define:

  • The Items to be processed
  • The function that processes a single Item

processing-chain will take care of spawning the process across all Items via parallelization. The user can also provide some extra processing configuration information (e.g., overwrite).

Highlights

  • Set-up generic data processing chains

Define the Items

Using a JSON file

[
    {
        "name": "item_1",
        "input_item_paths": ["test_1.npy", "test_2.npy", "test_2.npy"],
        "output_item_paths": ["output_1.nc"]
    },
    {
        "name": "item_2",
        "input_item_paths": ["test_1.npy", "test_2.npy"],
        "output_item_paths": ["output_2.nc"]
    },
    {
        "name": "item_3",
        "input_item_paths": ["test_6.npy", "test_7.npy", "test_8.npy"],
        "output_item_paths": ["output_3.nc"]
    }
]

Write the _process_item function

In rust:

fn _process_item(item: &Item) -> Result<bool> {
    // define how to process a single item
    println!(
        "Processing {} {:?} -> {:?}",
        item.name, item.input_item_path, item.output_item_path
    );
    // ...

    Ok(true)
}

If your function is written in Python and you don't feel like converting it to Rust (yet), you could use the inline-python crate.

use inline_python::python;

fn _process_item(item: &Item) -> Result<bool> {
    // define how to process a single item
    python! {
        print("Processing {} {} -> {}".format('item.name, item.input_item_path, item.output_item_path))
	};
    // ...

    Ok(true)
}

Some examples can be found here.

Dependencies

~6–17MB
~199K SLoC