run shell statements concurrently
Runs shell statements in parallel in the default shell. The default shell is
/bin/sh on POSIX, and on Windows it’s
Where the cmds step runs programs or executables,
passes the commands through to the system shell. This means all your usual shell
expressions are available, such as ~ expansions and your favorite bashisms.
If you just want to run parallel programs, scripts or executables with
arguments, you do NOT need to use
shells, you can use pypyr.steps.cmds instead. This will incur less processing overhead, because it
won’t have to instantiate the shell first.
You do NOT need shell to run a console-based executable or a script file (like .sh, .bat, .ps1), you can just use cmds instead.
Step input can take two forms: simple syntax or expanded syntax. Simple syntax is just a list of strings. This will run the shell statements in parallel with the default options.
# simple syntax - name: pypyr.steps.shells comment: list of simple cmd strings to run concurrently in: cmds: - echo one - echo two
If you want to override any of the default run options, you can use expanded syntax instead:
# expanded syntax # when save: True - name: pypyr.steps.shells in: cmds: run: - echo one - echo two save: True cwd: ./ bytes: False encoding: utf-8 # when save: False - name: pypyr.steps.shells in: cmds: run: - echo one - echo two cwd: .. stdout: ./path/out.txt stderr: ./path/err.txt append: False
run is mandatory in expanded syntax. All other inputs are optional. The
example shows which options are relevant depending on whether
False by default.
All inputs support substitutions. This means you can inject variables into any of the step inputs - see here for an example using the cmds step cmds variable substitution - you can do the exact same thing here with the shells step.
You can combine simple and expanded syntax - any of the top-level list items can be in expanded syntax to specify extra options:
- name: pypyr.steps.shells in: cmds: - echo one - run: echo two save: True cwd: ./ bytes: False encoding: utf-8 - run: - echo three - echo four cwd: ../ stdout: ./path/out.txt stderr: ./path/err.txt append: False - echo five
In this example all the commands will start at the same time and run in parallel. In other words, one, two, three, four & five will run concurrently.
Note that each run instruction executes in its own shell session. If you instead
want to run multiple shell statements in the same shell session, see
multiple inline shell statements
for details - the same principles of the serial
shell step apply here to the
If you want to run a sequence of shell statements serially (i.e one after the after), see pypyr.steps.shell instead. You can also embed serial sequences in your parallel workload - see the next section combine parallel & serial execution for details.
combine parallel & serial execution
If you want to start some shell statements at the same time, but still create a serial sequence where some commands run in serial one after the other, you can use a nested list to encode a serial sequence as part of the parallel workload:
- name: pypyr.steps.shells in: cmds: - echo A - [echo B.1, echo B.2, echo B.3] - echo C
This example will run as follows:
A B.1 -> B.2 -> B3 C
- A, B.1 & C will all start concurrently.
- B.2 will only start after B.1 completes successfully.
- B.3 will only start after B.2 completes successfully.
In a serial sequence, subsequent commands will only run when the previous command completes successfully. B.2 and B.3 will NOT run if B.1 raised an error.
You can specify the serial sequence sub-list in the pipeline using
[cmd1, cmd2] square brackets, or by using nested list syntax as per usual in yaml:
- name: pypyr.steps.shells in: cmds: - echo A - - echo B.1 - echo B.2 - echo B.3 - echo C
You can do exactly the same thing in expanded syntax - all these are equivalent ways of achieving the same result:
- name: pypyr.steps.shells in: cmds: run: - echo A - [echo B.1, echo B.2, echo B.3] - echo C save: True - name: pypyr.steps.shells in: cmds: run: - echo A - - echo B.1 - echo B.2 - echo B.3 - echo C cwd: mydir/subdir
capture shell output
Refer to cmds save output to context for details on how to capture stdout & stderr output and use it in subsequent steps in the pipeline.
cmds you can control how to parse the results with the optional
Be aware that
cmdOut.cmd will be the original input shell statement as a
string, it will not be split into a list of its component arguments like it does
save output to file
Refer to cmds save output to file for details on how to use the following optional inputs:
These inputs allow you to output to a file, or to redirect stderr to stdout.
redirect output to null device
See redirect cmd output to null device for details on how to redirect the shell’s output to system null.
The error handling principles are identical to that documented for asynchronous parallel cmds error handling.
See the documentation for the serial
shell step for details: