#run-command #parallel #tokio #cli #command-line-interface #progress-bar #command-line-utilities

app rust-parallel

Fast command line app in rust/tokio to run commands in parallel. Similar interface to GNU parallel or xargs.

74 releases (35 stable)

1.17.0 Mar 9, 2024
1.16.0 Feb 16, 2024
1.12.0 Dec 22, 2023
1.11.0 Nov 25, 2023
0.1.11 Nov 29, 2022

#39 in Asynchronous

Download history 13/week @ 2023-12-18 2/week @ 2024-01-01 16/week @ 2024-01-08 19/week @ 2024-01-15 73/week @ 2024-01-22 58/week @ 2024-01-29 30/week @ 2024-02-05 181/week @ 2024-02-12 74/week @ 2024-02-19 217/week @ 2024-02-26 830/week @ 2024-03-04 136/week @ 2024-03-11 61/week @ 2024-03-18

1,251 downloads per month

MIT license

155KB
2.5K SLoC

Rust 2K SLoC // 0.0% comments Shell 358 SLoC // 0.0% comments

rust-parallel

Crates.io CI workflow

Run commands in parallel and aggregate outputs. Async application using tokio.

Example commands and detailed manual.

Listed in Awesome Rust - utilities

Similar interface to GNU Parallel or xargs plus useful features:

Contents:

Installation:

Recommended:

  1. Download a pre-built release from Github Releases for Linux, MacOS, or Windows.
  2. Extract the executable and put somewhere in your $PATH.

For manual installation/update:

  1. Install Rust
  2. Install the latest version of this app from crates.io:
$ cargo install rust-parallel   
  1. The same cargo install rust-parallel command will also update to the latest version after initial installation.

Documents:

  1. Examples - complete runnable commands to give an idea of overall features.
  2. Manual - more detailed manual on how to use individual features.
  3. Benchmarks
  4. Output Interleaving - output interleaving in rust-parallel compared with other commands.

Tech Stack:

  • anyhow used for application error handling to propogate and format fatal errors.
  • clap command line argument parser.
  • itertools using multi_cartesian_product to process ::: command line inputs.
  • indicatif optional TUI progress bar.
  • regex optional regular expression capture groups processing for -r/--regex option.
  • tokio asynchronous runtime for rust. From tokio this app uses:
    • async / await functions (aka coroutines)
    • Singleton CommandLineArgs instance using tokio::sync::OnceCell.
    • Asynchronous command execution using tokio::process::Command
    • tokio::sync::Semaphore used to limit number of commands that run concurrently.
    • tokio::sync::mpsc::channel used to receive inputs from input task, and to send command outputs to an output writer task. To await command completions, use the elegant property that when all Senders are dropped the channel is closed.
  • tracing structured debug and warning logs.
  • which used to resolve command paths for path cache.

Dependencies

~10–23MB
~326K SLoC