15 releases
0.2.0 | Sep 9, 2024 |
---|---|
0.1.14 | Jan 12, 2024 |
0.1.12 | Apr 12, 2023 |
0.1.11 | Dec 24, 2022 |
0.1.2 | Apr 30, 2021 |
#62 in Concurrency
164 downloads per month
16KB
323 lines
parallel-sh
parallel-sh
was heavily inspired by Rust Parallel (parallel) parallelizing 'otherwise non-parallel command-line tasks.' But instead of trying to recreate the full functionality of GNU Parallel parallel-sh
will simply execute (lines of) commands in the platform's preferred shell (by default 'sh -c' on Unix systems, and 'powershell.exe -c' on Windows) in separate threads.
What to expect:
- Output (stdout and stderr) of each child process is stored and printed only after the child exits.
- There is some simple logging and some runtime metric (via -v, -vv or -vvv) available.
- The whole crate is tiny, <400 lines of code (a lot of it command line argument parsing), and can quickly be modified to meet more complex requirements.
What is not part of parallel-sh
:
- There are no replacement strings (e.g. '{}') or input tokens. Commands will be executed as provided by argument, file or via stdin.
- Command sources will not be 'linked'. Arguments will be processed by preference:
- If ARGS are found,
--file
option and stdin are ignored. - If
--file
is provided anything on stdin is ignored. - Only when there are no command arguments and no '--file' option is found, any lines on stdin are treated as commands to execute.
- If ARGS are found,
- Stdin is not inherited from the parent and any attempt by the child processes to read from the stdin stream will result in the stream immediately closing. But you can use pipes, redirects etc. within each thread as long as your shell provides the functionality, e.g.
parallel-sh 'ls -1 |wc -l
orparallel-sh.exe "Get-ChildItem -Path * | Measure-Object -Line"
Most of the effects of these features can be achieved by processing the commands before passing them to parallel-sh
.
Options
parallel-sh 0.1.14
Execute commands in parallel
Usage: parallel-sh [OPTIONS] [clijobs]...
Arguments:
[clijobs]...
Options:
-q, --quiet Do not print `parallel-sh` warnings
-n, --dry-run Perform a trial run, only print what would be done (with -vv)
-v, --verbose... Sets the level of verbosity
-l, --log <FILE> Log output to file
--halt-on-error Stop execution if an error occurs in any thread
-j, --jobs <THREADS> Number of parallel executions
-s, --shell <SHELL> Shell to use for command execution. Must support '-c' (defaults to sh)
--no-shell Do not pass commands through a shell, but execute them directly
-f, --file <FILE> Read commands from file (one command per line)
-h, --help Print help
-V, --version Print version
Note
Per default commands are executed via -c "command", therefore the provided shell must support the '-c' option.
With --no-shell
the commands are started without passing them through a shell. This will avoid the overhead of starting a shell in each thread, but you will lose features like quotes, escaped characters, word splitting, glob patterns, variable substitution, etc.
The commands inherit parallel-sh
’s working directory.
Preference
-
Pass commands as arguments:
parallel-sh "sleep 2 && echo first" "sleep 1 && echo second"
-
Pass a file with one command (-line) per line:
parallel-sh -f /tmp/commands $ cat /tmp/commands sleep 2 && echo first sleep 1 && echo second
-
Pass commands via stdin:
echo -e 'sleep 2 && echo first\nsleep 1 && echo second' |parallel-sh
Dependencies
~1–7.5MB
~56K SLoC