#cli #parallel #tokio

app rust-parallel

Execute commands in parallel

30 releases

new 0.3.0 Mar 18, 2023
0.2.13 Mar 18, 2023
0.2.8 Feb 25, 2023
0.2.5 Jan 28, 2023
0.1.11 Nov 29, 2022

#51 in Asynchronous

Download history 120/week @ 2022-11-28 125/week @ 2022-12-05 56/week @ 2022-12-12 43/week @ 2022-12-19 110/week @ 2022-12-26 41/week @ 2023-01-02 50/week @ 2023-01-09 34/week @ 2023-01-16 27/week @ 2023-01-23 40/week @ 2023-01-30 40/week @ 2023-02-06 30/week @ 2023-02-13 66/week @ 2023-02-20 35/week @ 2023-02-27 23/week @ 2023-03-06 56/week @ 2023-03-13

180 downloads per month

Apache-2.0

20KB
325 lines

rust-parallel

Command-line utility to execute commands in parallel and aggregate their output.

Similar interface to GNU Parallel or xargs but implemented in rust and tokio.

Always executes 1 process for each input line, similar to xargs -n1 or -L1 options.

Being written in asynchronous rust it is quite fast - see benchmarks.

Crates.io

Contents:

Usage:

$ rust-parallel --help
Execute commands in parallel

By Aaron Riekenberg <aaron.riekenberg@gmail.com>

https://github.com/aaronriekenberg/rust-parallel
https://crates.io/crates/rust-parallel

Usage: rust-parallel [OPTIONS] [COMMAND_AND_INITIAL_ARGUMENTS]...

Arguments:
  [COMMAND_AND_INITIAL_ARGUMENTS]...
          Optional command and initial arguments to run for each input line

Options:
  -i, --input <INPUT>
          Input file or - for stdin.  Defaults to stdin if no inputs are specified

  -j, --jobs <JOBS>
          Maximum number of commands to run in parallel, defauts to num cpus

          [default: 8]

  -0, --null-separator
          Use null separator for reading input instead of newline

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

Installation:

Recommended:

  1. Download a pre-built release from Github Releases.
  2. Extract the executable and put somewhere in your $PATH.

For manual installation/update:

  1. Install Rust
  2. Install the latest version of this app from crates.io:
$ cargo install rust-parallel   
  1. The same cargo install rust-parallel command will also update to the latest version after initial installation.

Demos:

  1. Small demo of 5 echo commands. With -j5 all 5 commands are run in parallel. With -j1 commands are run sequentially:
$ cat >./test <<EOL
echo hi
echo there
echo how
echo are
echo you
EOL

$ cat test | rust-parallel -j5
are
hi
there
how
you

$ cat test | rust-parallel -j1
hi
there
how
are
you
  1. Specifying command and intial arguments on command line:
$ head -100 /usr/share/dict/words | rust-parallel md5 -s
MD5 ("aal") = ff45e881572ca2c987460932660d320c
MD5 ("A") = 7fc56270e7a70fa81a5935b72eacbe29
MD5 ("aardvark") = 88571e5d5e13a4a60f82cea7802f6255
MD5 ("aalii") = 0a1ea2a8d75d02ae052f8222e36927a5
MD5 ("aam") = 35c2d90f7c06b623fe763d0a4e5b7ed9
MD5 ("aa") = 4124bc0a9335c27f086f24ba207a4912
MD5 ("a") = 0cc175b9c0f1b6a831c399e269772661
MD5 ("Aani") = e9b22dd6213c3d29648e8ad7a8642f2f
MD5 ("Aaron") = 1c0a11cc4ddc0dbd3fa4d77232a4e22e
MD5 ("aardwolf") = 66a4a1a2b442e8d218e8e99100069877
  1. Using awk to form complete commands:
$ head -100 /usr/share/dict/words | awk '{printf "md5 -s %s\n", $1}' | rust-parallel
MD5 ("Abba") = 5fa1e1f6e07a6fea3f2bb098e90a8de2
MD5 ("abaxial") = ac3a53971d52d9ce3277eadf03f13a5e
MD5 ("abaze") = 0b08c52aa63d947b6a5601ee975bc3a4
MD5 ("abaxile") = 21f5fc27d7d34117596e41d8c001087e
MD5 ("abbacomes") = 76640eb0c929bc97d016731bfbe9a4f8
MD5 ("abbacy") = 08aeac72800adc98d2aba540b6195921
MD5 ("Abbadide") = 7add1d6f008790fa6783bc8798d8c803
MD5 ("abb") = ea01e5fd8e4d8832825acdd20eac5104
  1. Using as part of a shell pipeline. stdout and stderr from each command run are copied to stdout/stderr of the rust-parallel process.
$ head -100 /usr/share/dict/words | rust-parallel md5 -s | grep -i abba
MD5 ("Abba") = 5fa1e1f6e07a6fea3f2bb098e90a8de2
MD5 ("abbacomes") = 76640eb0c929bc97d016731bfbe9a4f8
MD5 ("abbacy") = 08aeac72800adc98d2aba540b6195921
MD5 ("Abbadide") = 7add1d6f008790fa6783bc8798d8c803
  1. Working on a set of files from find command. The -0 option works nicely with find -print0 to handle filenames with newline or whitespace characters:
$ mkdir testdir

$ touch 'testdir/a b' 'testdir/b c' 'testdir/c d'

$ find testdir -type f -print0 | rust-parallel -0 gzip -f -k

$ ls testdir
'a b'  'a b.gz'  'b c'  'b c.gz'  'c d'  'c d.gz'
  1. By default rust-parallel reads input from stdin only. The -i option can be used 1 or more times to override this behavior. -i - means read from stdin, -i ./test means read from the file ./test:
$ cat >./test <<EOL
foo
bar
baz
EOL

$ head -5 /usr/share/dict/words | rust-parallel -i - -i ./test echo
A
aalii
aa
a
aal
bar
foo
baz
  1. Set environment variable RUST_LOG=debug to see debug output.
$ head -10 /usr/share/dict/words | RUST_LOG=debug rust-parallel md5 -s

Benchmarks:

See the wiki page for benchmarks.

Features:

  • Use only safe rust.
  • Use only asynchronous operations supported by tokio, do not use any blocking operations. This includes writing to stdout and stderr.
  • Support arbitrarily large number of input lines, avoid O(number of input lines) memory usage. In support of this:
    • tokio::sync::Semaphore is used carefully to limit the number of commands that run concurrently. Do not spawn tasks for all input lines immediately to limit memory usage.
  • Support running commands on local machine only, not on remote machines.

Tech Stack:

Dependencies

~5–10MB
~163K SLoC