run your first pipeline

run a built-in pipeline

Run one of the built-in pipelines to get a feel for it:

$ pypyr echo "Ceci n'est pas une pipe"

echo is the name of a built-in pypyr pipeline. The pipeline simply echoes the input string back to console output using the built-in echo step.

The actual pipeline looks like this:

# To execute this pipeline, shell something like:
# pypyr echo text goes here
context_parser: pypyr.parser.string
steps:
  - name: pypyr.steps.echo
    in:
      echoMe: '{argString}'

This a silly pipeline with only one step. It doesn’t do much other than show you a basic example as an introduction of what a basic pipeline looks like.

The pypyr.parser.string context_parser tells pypyr to concatenate all the input args you specify from the CLI and make the result available to the pipeline as a variable named argString. pypyr.steps.echo then uses a substitution expression to get the value of argString and print it out to the console.

You can achieve the same end-result by hard-coding the string you want to echo into the pipeline yaml rather than passing it in from the cli as a positional argument:

$ pypyr magritte

# ./magritte.yaml
# To execute this pipeline, shell something like:
# pypyr magritte
steps:
  - name: pypyr.steps.echo
    comment: output echoMe
    in:
      echoMe: Ceci n'est pas une pipe

write your first pipeline

A pipeline is simply a yaml file.

The simplest pipeline just needs a steps list. This is the default entry point. steps is a list that has a bunch of individual steps in it that will run sequentially one after the other.

The following examples will only have one or two steps in them, for the sake of an easy introduction, but you can mix and match as many steps as you need in your pipelines.

posix

# ./my-first-pipeline.yaml
# To execute this pipeline, shell something like:
# pypyr my-first-pipeline
steps:
  - name: pypyr.steps.echo
    comment: output echoMe
    in:
      echoMe: this is step 1

  - name: pypyr.steps.cmd
    comment: actually invokes any program on your system.
             the program must be available in current path.
             run any program you fancy.
             we're just running /bin/echo here for an easy demo.
    in:
      cmd: echo this is step 2

windows

# ./my-first-pipeline.yaml
# To execute this pipeline, shell something like:
# pypyr my-first-pipeline
steps:
  - name: pypyr.steps.echo
    comment: output echoMe
    in:
      echoMe: this is step 1

  - name: pypyr.steps.cmd
    comment: Actually invokes any executable program on your system.
             The program must be available in current path.
             Run any program you fancy.
             Here we're running the powershell executable.
             We're just using powershell's echo for an easy demo,
             but you can invoke any powershell command you like instead.
    in:
      cmd: powershell.exe -NoProfile -Command "echo 'this is step 2'"

Save this file as ./my-first-pipeline.yaml.

Now you can run your pipeline like this:

term

$ pypyr my-first-pipeline
this is step 1
this is step 2

$

The second step here is the cmd step - this built-in step allows you to run any executable available on your system. For the sake of hello-world simplicity, we’re just running good old echo, but you can substitute this with any executable program available in your $PATH.

If you don’t want to rely on $PATH to resolve your program location, you can specify the full absolute path instead. You can also specify a relative path to the current working directory.

We’re starting with pypyr.steps.cmd because you’re likely to use it a lot. But pypyr also has a whole bunch of other ready-made built-in steps that you can use for various tasks such as working with files, variables, merging configuring, paths & custom inline code. You can also make your own custom step without fuss.

cross-platform compatibility

Linux & MacOS can run the posix sample yaml as is.

On Windows, if you’re running pypyr via WSL you can also use the posix samples as is, but if you’re running pypyr natively on Windows use the windows sample yaml instead. The reason for this is that posix-compliant systems have /bin/echo available as a discrete program, whereas on Windows echo is baked into the shell instead.

Just for the purposes of an introductory demonstration of running an .exe on Windows via pypyr.steps.cmd, we’re invoking powershell.exe and running the echo command inside it. In real world scenarios, if you did really want to run a shell command, you could just use pypyr.steps.shell instead.

In general, pypyr pipelines can absolutely be cross-platform. In fact, this is one of the reasons you might be using pypyr. For example, generally you can specify all filesystem paths with / as you would for POSIX, and this will automatically resolve correctly to the Windows \.

passing arguments to your pipeline

You can pass arguments from the cli to your pipeline easy peasy.

All you need to do is add a context_parser.

In this example we’ll be using a key-value pair parser, which will let you specify your own key-value pairs so you can access the values you need by key. There are other types of built-in context parser that can parse the cmd input differently for you.

The context parser will add these values into the pypyr context for you. Your pipeline steps can then use these values as they please with {formatting expressions}.

This time round, instead of running /bin/echo, we’re going to execute the ping program:

posix

# ./pipe-with-args.yaml
# To execute this pipeline, shell something like:
# pypyr pipe-with-args akey=avalue anotherkey="another arbitrary value" count=1 ip=127.0.0.1
context_parser: pypyr.parser.keyvaluepairs
steps:
  - name: pypyr.steps.echo
    in:
      echoMe: this is akey's value - {akey}. It came from the cli args. {anotherkey}.

  - name: pypyr.steps.cmd
    in:
      cmd: ping -c {count} {ip}

windows

# ./pipe-with-args.yaml
# To execute this pipeline, shell something like:
# pypyr pipe-with-args akey=avalue anotherkey="another arbitrary value" count=1 ip=127.0.0.1
context_parser: pypyr.parser.keyvaluepairs
steps:
  - name: pypyr.steps.echo
    in:
      echoMe: this is akey's value - {akey}. It came from the cli args. {anotherkey}.

  - name: pypyr.steps.cmd
    comment: runs C:\Windows\System32\PING.EXE
    in:
      cmd: ping /n {count} {ip}

Save this file as ./pipe-with-args.yaml.

Now you can run this pipeline like this:

term

$ pypyr pipe-with-args akey=avalue anotherkey="another arbitrary value" count=1 ip=127.0.0.1
this is akey's value - avalue. It came from the cli args. another arbitrary value.
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.063 ms

--- 127.0.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.063/0.063/0.063/0.000 ms

$

branching, loops, retries

You can add extra functionality to any step in pypyr by decorating the step with a decorator. pypyr gives you built-in decorators to loop, retry a failure condition automatically and selectively run a step only if a certain condition is true - all of this without you having to code it yourself.

conditional execution

You can dynamically run or skip any step you want:

# ./conditional-logic.yaml
context_parser: pypyr.parser.keys
steps:
  - name: pypyr.steps.default
    comment: set default values for optional cli inputs
    in:
      defaults:
        A: False
        B: False

  - name: pypyr.steps.echo
    in:
      echoMe: begin

  - name: pypyr.steps.echo
    run: '{A}'
    in:
      echoMe: A

  - name: pypyr.steps.echo
    run: '{B}'
    in:
      echoMe: B

  - name: pypyr.steps.echo
    in:
      echoMe: end

The pypyr.parser.keys context parser creates a boolean variable set to True for every key you pass in via the cli.

The run condition on the A and B steps uses a {formatting expression} to check whether the boolean value of A or B is True.

You can now selectively control which steps you want to run by changing the cli input arguments:

$ pypyr conditional-logic
begin
end

$ pypyr conditional-logic A
begin
A
end

$ pypyr conditional-logic B
begin
B
end

$ pypyr conditional-logic A B
begin
A
B
end

Notice that we’re just using the pypyr.steps.echo step here to demonstrate the principle - you can apply the conditional run decorator to any given step, including your own custom commands.

automatic retry

You can automatically retry any given step if it fails, without having to write any code.

steps:
  - name: pypyr.steps.cmd
    retry:
      max: 4
    in:
      cmd: curl https://arb-unreliable-url-example-xyz/

This is calling the curl program - we’re just using this because it’s widely available on different operating systems, but you could use wget or whatever takes your fancy instead. If curl returns an error, pypyr will retry the command automatically up to 4 times.

If the 4th retry still fails, pypyr will not go to the next step. pypyr will raise the error & report failure instead.

static loop

Let’s add a simple loop to a command using the foreach decorator:

posix

# ./fruit-loop.yaml
steps:
  - name: pypyr.steps.echo
    in:
      echoMe: begin

  - name: pypyr.steps.cmd
    foreach: [apple, pear, banana]
    in:
      cmd: echo arb-command --arg {i}

  - name: pypyr.steps.echo
    in:
      echoMe: end

windows

# ./fruit-loop.yaml
steps:
  - name: pypyr.steps.echo
    in:
      echoMe: begin

  - name: pypyr.steps.cmd
    foreach: [apple, pear, banana]
    in:
      cmd: powershell.exe -NoProfile -Command "echo 'arb-command --arg {i}'"

  - name: pypyr.steps.echo
    in:
      echoMe: end

Notice that for the cmd step we’re just using echo for the sake of example, you can instead use whatever executable is available on your system with whatever input args & switches you want.

Just putting echo in front of a command is a handy way of troubleshooting more complex command sequences - you print out exactly the final formatted value pypyr passes to the underlying operating system.

{i} is a context variable that contains the current item of the iterator.

Running this pipeline looks like this:

term

$ pypyr fruit-loop
begin
arb-command --arg apple
arb-command --arg pear
arb-command --arg banana
end

$

dynamic loop cli input args

Of course, you don’t have to hard-code the list the you want to iterate. Let’s inject the values from the cli. As always, use a context parser to access cli arguments: in this case, pypyr.parser.list will parse all the input arguments into a list named {argList} for you.

posix

# ./fruit-loop
context_parser: pypyr.parser.list
steps:
  - name: pypyr.steps.echo
    in:
      echoMe: begin

  - name: pypyr.steps.cmd
    foreach: '{argList}'
    in:
      cmd: echo arb-command --arg {i}

  - name: pypyr.steps.echo
    in:
      echoMe: end

windows

# ./fruit-loop
context_parser: pypyr.parser.list
steps:
  - name: pypyr.steps.echo
    in:
      echoMe: begin

  - name: pypyr.steps.cmd
    foreach: '{argList}'
    in:
      cmd: powershell.exe -NoProfile -Command "echo 'arb-command --arg {i}'"

  - name: pypyr.steps.echo
    in:
      echoMe: end

Notice that the foreach input now uses a {formatting expression} to get the value of the argList variable set by the cli args input context parser, instead of a hard-coded list.

Now you can control dynamically how many to times to run your command from the cli like this:

term

$ pypyr fruit-loop
begin
end

$ pypyr fruit-loop apple pear
begin
arb-command --arg apple
arb-command --arg pear
end

The first run, without any arguments, just prints begin and end - since the input list is empty there is nothing over which to loop.

what next?

You probably should check out:

understanding basic pipeline structure
How to declare & use variables
How to do branching with conditional logic
How to use loops
Deal with errors

run your first pipeline permalink

run a built-in pipeline permalink

write your first pipeline permalink

cross-platform compatibility permalink

passing arguments to your pipeline permalink

branching, loops, retries permalink

conditional execution permalink

automatic retry permalink

static loop permalink

dynamic loop cli input args permalink

what next? permalink

run your first pipeline

run a built-in pipeline

write your first pipeline

cross-platform compatibility

passing arguments to your pipeline

branching, loops, retries

conditional execution

automatic retry

static loop

dynamic loop cli input args

what next?