1. pypyr
  2. docs
  3. getting started
  4. error handling

error handling permalink

stop all processing on error permalink

pypyr runs pipelines. . . and a pipeline is a sequence of steps. By default subsequent steps in the sequence should not run if a previous step failed.

If your desired behavior is for pipeline processing to stop and subsequent steps NOT to run once an error occurs somewhere, you don’t have to do anything special, because this is what pypyr does by default.

steps:
  - name: pypyr.steps.echo
    in:
      echoMe: A
  - name: pypyr.steps.py
    comment: deliberately raise arbitrary error
    in:
      py: raise ValueError('arb')
  - name: pypyr.steps.echo
    comment: this step will never run,
             because the previous step
             always fails.
    in:
      echoMe: unreachable

ignore error on a step permalink

You can ignore an error on a specific step by setting the swallow step decorator to True.

steps:
  - name: pypyr.steps.echo
    in:
      echoMe: A
  - name: pypyr.steps.py
    swallow: True
    in:
      py: raise ValueError('arb')
  - name: pypyr.steps.echo
    in:
      echoMe: You'll see me, because you told pypyr to swallow the error in the previous step.

retry on error permalink

You can make a any step retry automatically on error by setting the retry step decorator.

steps:
  - name: pypyr.steps.cmd
    comment: automatically retry curl to an unreliable url.
             pipeline proceeds with next step as soon as curl succeeds.
             will retry 4X with 0.5s sleep between retries.
             if the 4th retry still fails, raise error & report failure.
    retry:
      max: 4
      sleep: 0.5
    in:
      cmd: curl https://arb-unreliable-url-example/

failure handlers permalink

A failure handler is an optional step-group that pypyr jumps to when an error happens. In traditional programming terms, it’s pretty much a catch block. Once the failure handler completes, the pipeline exits reporting failure.

pypyr looks for a failure handler if swallow on the failing step is False and only after any retry on the step exhausts.

Any given step-group can be a failure handler. If you don’t explicitly specify a failure handler pypyr will look for a step group named on_failure. If on_failure doesn’t exist it doesn’t matter, pypyr will still quit the pipeline reporting the original error.

# ./failure-handler-example.yaml
steps:
  - name: pypyr.steps.echo
    in:
      echoMe: A
  - name: pypyr.steps.py
    in:
      py: raise ValueError('arb')
  - name: pypyr.steps.echo
    comment: this step will never run,
            because the previous step
            always fails.
    in:
      echoMe: unreachable

on_failure:
  - name: pypyr.steps.echo
    in:
      echoMe: B

Running this pipeline will result in:

$ pypyr failure-handler-example
A
Error while running step pypyr.steps.py at pipeline yaml line: 6, col: 5
Something went wrong. Will now try to run on_failure.
B

ValueError: arb

$ echo $?
255

If a failure handler encounters another exception while processing an exception, then pypyr will log and report both that exception and the original cause’s exception, although the original cause exception is what pypyr highlights on CLI exit and raises to API consumers.

set your own failure handler permalink

You can set any given step-group to be a failure handler.

Similarly to if on_failure does not exist, pypyr will not raise an additional error if the specified custom failure-handler doesn’t exist. The log output will tell you that it did look for it but couldn’t find it.

# ./my-pipeline.yaml
steps:
  - name: pypyr.steps.echo
    in:
      echoMe: A
  - name: pypyr.steps.py
    in:
      py: raise ValueError('arb')
  - name: pypyr.steps.echo
    in:
      echoMe: unreachable

arb_group:
  - name: pypyr.steps.echo
    in:
      echoMe: B

from the cli permalink

When you invoke pypyr from the cli and you don’t want to use the default on_failure failure handler you can use the --failure input argument:

$ pypyr my-pipeline --failure arb_group

when calling a pipeline permalink

When you call a pipeline from within another pipeline using
pype you can override the default on_failure by setting your own:

- name: pypyr.steps.pype
  in:
    pype:
      name: my-pipeline
      failure: arb_group

within a pipeline permalink

You can specify your own failure-handler when you call, jump or switch to another step-group in your pipeline.

# ./call-with-failure-group.yaml
steps:
  - name: pypyr.steps.echo
    in:
      echoMe: A
  - name: pypyr.steps.call
    in:
      call:
        groups: call_me
        failure: arb_group
  - name: pypyr.steps.echo
    in:
      echoMe: unreachable

call_me:
  - name: pypyr.steps.echo
    in:
      echoMe: B
  - name: pypyr.steps.assert
    in:
      assert: False
  - name: pypyr.steps.echo
    in:
      echoMe: unreachable

arb_group:
  - name: pypyr.steps.echo
    in:
      echoMe: C

Notice that when you run this pipeline, when the arb_group failure handler completes reporting failure, pypyr still falls back to processing the actual pipeline’s handler, if it exists. In this case, because we don’t invoke the pipeline with a custom failure handler set, it defaults to looking for on_failure.

This is the same concept as an error bubbling up to the top of the stack.

$ pypyr call-with-failure-group
A
B
Error while running step pypyr.steps.assert at pipeline yaml line: 19, col: 5
Something went wrong. Will now try to run arb_group.
C
Error while running step pypyr.steps.call at pipeline yaml line: 6, col: 5
Something went wrong. Will now try to run on_failure.

AssertionError: assert False evaluated to False.
$

call or jump in failure handler permalink

You can call or jump with a failure handler. This is handy if you have shared or common code you want to run on different error and success conditions, because you can encapsulate your failure handling code in its own step-group. A very typical scenario for this is if you want to send a notification on both success and failure.

# ./call-on-failure.yaml
steps:
  - name: pypyr.steps.assert
    in:
      assert: False
  - name: pypyr.steps.echo
    in:
      echoMe: unreachable

sg1:
  - name: pypyr.steps.echo
    in:
      echoMe: B
  - name: pypyr.steps.call
    in:
      call: sg2
  - name: pypyr.steps.echo
    in:
      echoMe: D

sg2:
  - name: pypyr.steps.echo
    in:
      echoMe: C

on_failure:
  - name: pypyr.steps.echo
    in:
      echoMe: A
  - name: pypyr.steps.call
    in:
      call: sg1
  - name: pypyr.steps.echo
    in:
      echoMe: E

Running this pipeline results in:

$ pypyr call-on-failure
Error while running step pypyr.steps.assert at pipeline yaml line: 3, col: 5
Something went wrong. Will now try to run on_failure.
A
B
C
D
E

AssertionError: assert False evaluated to False.

Notice that the called group can also call another group.

If you jump from a failure handler, you’re still within the execution context of the failure handler. This means that once the group you jump to completes, pypyr will still quit reporting failure, unless you use one of the Stop instructions somewhere in the execution chain.

don’t quit pipeline reporting failure permalink

Once a pipeline’s failure handler completes, by default pypyr quits the pipeline reporting failure. If you want to handle the error and quit the pipeline reporting success, you can use a Stop instruction to prevent the failure handler from completing and raising the error.

Depending on whether you want to stop pypyr entirely, or only the pipeline, or only the currently called or jumped to group, you can use any of stop, stoppipeline or stopstepgroup.

# ./stop-on-failure.yaml
steps:
  - name: pypyr.steps.assert
    in:
      assert: False
  - name: pypyr.steps.echo
    in:
      echoMe: unreachable

on_failure:
  - name: pypyr.steps.echo
    in:
      echoMe: A
  - pypyr.steps.stop
  - name: pypyr.steps.echo
    in:
      echoMe: unreachable

When you run this pipeline pypyr will quit reporting success because the failure handler considers the error condition as handled when you issue the stop instruction:

$ pypyr stop-on-failure
Error while running step pypyr.steps.assert at pipeline yaml line: 3, col: 5
Something went wrong. Will now try to run on_failure.
A

$ echo $?
0

If you use call or jump inside your failure handler, you can issue the appropriate Stop instruction in the groups that you invoke.

use runtime error details inside pipeline permalink

pypyr saves all run-time errors to a list in context called runErrors.

runErrors:
  - name: Error Name Here
    description: Error Description Here
    customError: # whatever you put into onError on step definition
    line: 1 # line in pipeline yaml for failing step
    col: 1 # column in pipeline yaml for failing step
    step: my.bad.step.name # failing step name
    exception: ValueError('arb') # the actual python error object
    swallowed: False # True if err was swallowed

The last error will be the last item in the list. The first error will be the first item in the list.

using error information in subsequent steps permalink

You can use runErrors in your step-group’s failure handler, or if you set swallow=True on the failing step you can use runErrors in subsequent steps to use the actual error information.

# ./use-error-info
steps:
  - name: pypyr.steps.echo
    in:
      echoMe: A
  - name: pypyr.steps.assert
    swallow: True
    in:
      assert: False
  - name: pypyr.steps.echo
    in:
      echoMe: there was a problem on line {runErrors[0][line]}
  - name: pypyr.steps.py
    in:
      py: raise ValueError('arb')
  - name: pypyr.steps.echo
    in:
      echoMe: unreachable

on_failure:
  - name: pypyr.steps.echo
    in:
      echoMe: B
  - name: pypyr.steps.assert
    in:
      assert:
        this: '{runErrors[0][name]}'
        equals: AssertionError
  - name: pypyr.steps.assert
    in:
      assert:
        this: '{runErrors[1][name]}'
        equals: ValueError
  - name: pypyr.steps.assert
    in:
      assert:
        this: '{runErrors[1][description]}'
        equals: arb

Notice this pipeline uses runErrors information both in the steps group after the swallow, and also in the on_failure handler. The asserts in the failure handler effectively tests if the errors are as expected.

$ pypyr use-error-info
A
pypyr.steps.assert Ignoring error because swallow is True for this step.
AssertionError: assert False evaluated to False.
there was a problem on line 6
Error while running step pypyr.steps.py at pipeline yaml line: 14, col: 5
Something went wrong. Will now try to run on_failure.
B

ValueError: arb

add extra error information permalink

You can provide your own additional description or even your own object to add to the error by using the onError decorator.

- name: pypyr.steps.assert
  comment: deliberately raise error
           with custom error object
  in:
    assert: False
  onError:
    myerr_code: 123
    myerr_description: "my err description"

This information is available in the runErrors list in context.

raise a custom error permalink

You can raise your own errors in pypyr. At easiest, use the built-in assert, but you can also raise any Python built-in error.

assertion error permalink

You can use assert to raise an error if the assertion condition fails.

steps:
  - name: pypyr.steps.assert
    in:
      assert: False
  - name: pypyr.steps.echo
    comment: this step won't ever run because pipeline always
             fails on previous step.
    in:
      echoMe: unreachable
Remember you can use formatting substitutions for the assert condition selectively to raise the AssertionError or not.

custom error permalink

You can raise any built-in Python exception from an inline python step.

You can also raise pypyr’s built-in errors or your own custom error objects if you import the appropriate module as part of the py step, per usual Python module import referencing.

steps:
  - name: pypyr.steps.echo
    in:
      echoMe: begin
  - name: pypyr.steps.py
    in:
      py: raise ValueError('arb error text here')
  - name: pypyr.steps.echo
    comment: this step won't ever run 
             because pipeline always
             fails on previous step.
    in:
      echoMe: unreachable

see also

last updated on .