topics

Browse pypyr pipeline runner technical documentation by Topic.

• args

  #

  How to pass your own arguments from the cli.

  pypyr.parser.argskwargs

  Pass args and/or key=value pairs from cli input to the pipeline.

  pypyr.parser.tomlfile

  Load toml file into context from a cli arg path input.

  custom args

  Passing custom args from the cli to your pipeline.

  pypyr.parser.yamlfile

  Put yaml file into context from a cli arg path input.

  pypyr.parser.string

  Concatenate all cli input args into single string argString.

  pypyr.parser.list

  Put each cli input arg into a list argList.

  pypyr.parser.keyvaluepairs

  Pass key-value pairs from cli input args to the pipeline.

  pypyr.parser.keys

  Boolean cli switches True if exist.

  pypyr.parser.jsonfile

  Put json file into context from a cli arg path input.

  pypyr.parser.json

  Put json input string from cli arg into context.

  pypyr.parser.dict

  Parse cli key=value pairs into argDict

• built-in summary tables

  #

  Convenient one page summary listing of all available items in a section.

  built-in pipelines

  Ready-made built-in pipelines

  context parsers

  pypyr context parsers initialize the context from your own custom cli input arguments.

  step decorators

  Selectively run, skip, repeat, loop & handle errors on individual steps.

  plugins overview

  A listing of pypyr’s official plug-ins. Add AWS & Slack functions to your pipelines.

  aws steps overview

  A listing of AWS built-in ready-made steps. Scroll through all pypyr’s AWS built-in steps here.

  steps overview

  A listing of pypyr built-in ready-made steps. Scroll through all pypyr’s built-in steps here.

  substitutions

  Use string formatting expressions to make substitutions or token replacements in context strings, fields or config.

• collections

  #

  Using collections like lists, sets & dicts in the pypyr context.

  pypyr.steps.add

  Add item to a set.

  pypyr.steps.append

  Append item to a list.

• context

  #

  The pypyr context maintains state for the duration of the pipeline. You use the context to persist & pass values between steps in the pipeline.

  variables

  How to use variables & arguments in a pipeline.

  pypyr.steps.configvars

  Inject variables from config into pipeline.

  pypyr.steps.add

  Add item to a set.

  pypyr.steps.append

  Append item to a list.

  pypyr.steps.set

  Set & format context keys.

  pypyr.steps.contextcopy

  Copy entire context keys.

  custom context parser

  Create your own context parser to parse cli input arguments.

  context parsers

  pypyr context parsers initialize the context from your own custom cli input arguments.

  in decorator

  Add arguments to context for the current step.

  pypyr.steps.default

  Set values if they do not exist already.

  pypyr.steps.contextsetf

  Old name for set. Use set instead.

  pypyr.steps.contextmerge

  Merges values into context, preserving the existing context hierarchy.

  pypyr.steps.contextclearall

  Wipe the entire context.

  pypyr.steps.contextclear

  Remove specified items from context.

• control-of-flow

  #

  Flow control for your pipelines. Conditional execution, loops, jump between steps & stop processing.

  conditional logic

  How to run or skip steps conditionally.

  loops

  How to loop or iterate over a step.

  pypyr.steps.switch

  Conditional branching for IF-THEN-ELSE control-of-flow.

  pypyr.steps.assert

  Stop pipeline if item in context is not as expected.

  pypyr.steps.call

  Call another step-group. Continue from the same place after the called groups complete.

  while loop decorator

  Repeat step until stop condition is True or up to the maximum iteration count.

  skip decorator

  Skip this step if condition True.

  run decorator

  Run this step only if condition True.

  foreach loop decorator

  Repeat step for each item in list.

  pypyr.steps.stopstepgroup

  Stop current step-group.

  pypyr.steps.stoppipeline

  Stop current pipeline.

  pypyr.steps.stop

  Stop pypyr entirely.

  pypyr.steps.pype

  Call another pipeline from the current pipeline.

  pypyr.steps.jump

  Jump to another step-group. The rest of the current step-group doesn’t run.

• custom code

  #

  pypyr is super easy to customize with very non-demanding function signatures for your own steps, context parsers & pipeline loaders. Call pipelines from your own code via the API.

  custom module name resolution

  How to reference custom modules in your pipelines.

  custom retry backoff

  Create your own custom retry backoff algorithm.

  api

  Easily extend pypyr with the simple Python API. Create your own steps and context argument parsers. Call pypyr programmatically from your own code.

  custom context parser

  Create your own context parser to parse cli input arguments.

  custom pype loader

  Create a custom pipeline loader to load pipelines from a different place.

  developer's guide for pypyr

  Handy developer & coding tips for pypyr contributors.

  run pipeline api

  Run pipeline programmatically from the API in your own code.

  custom step

  Create your own custom step.

• debug

  #

  Debug & troubleshoot your pipelines.

  pypyr.steps.assert

  Stop pipeline if item in context is not as expected.

  description decorator

  Descriptive text that prints to output when step runs.

  pypyr.steps.debug

  Pretty print context to output.

• environment variables

  #

  Use $ENV environment variables in your pipelines.

  pypyr.steps.envget

  Get $ENVs & use a default fallback if they don’t exist.

  pypyr.steps.env

  Get, set or unset $ENVs.

• error handling

  #

  pypyr stops execution on any error by default. You can also handle errors instead, and selectively trap, ignore or retry exceptions.

  error handling

  How to handle errors in your pipelines.

  retry decorator

  Retry step until it succeeds.

  swallow decorator

  Swallow step error & continue with pipeline.

  onError decorator

  Add custom error data on step error.

• execute external program

  #

  Run external programs, applications & commands in your pipeline.

  pypyr.steps.venv

  Create a virtual environment (venv).

  pypyr.steps.shells

  Run shell statements concurrently in parallel.

  pypyr.steps.cmds

  Run external programs, commands & scripts concurrently in parallel.

  pypyr.steps.python

  Get the absolute path of the current Python executable.

  pypyr.steps.shell

  Run commands in the default shell. Use for pipes, wildcards, $ENVs, ~

  pypyr.steps.cmd

  Run any external program, command, script.

• filesystem

  #

  Read & write files and work with paths on the local filesystem.

  pypyr.steps.fileread

  Read file into pypyr context.

  pypyr.steps.filewrite

  Write payload to file.

  pypyr.steps.filewritetoml

  Write payload to file in toml format.

  pypyr.steps.fetchtoml

  Load toml file into pypyr context.

  pypyr.parser.tomlfile

  Load toml file into context from a cli arg path input.

  pypyr.steps.fileformattoml

  Find & replace substitution {tokens} in a toml file.

  pypyr.parser.yamlfile

  Put yaml file into context from a cli arg path input.

  pypyr.parser.jsonfile

  Put json file into context from a cli arg path input.

  pypyr.steps.pathcheck

  Check if paths exist on filesystem.

  pypyr.steps.glob

  Get paths from glob expression.

  pypyr.steps.filewriteyaml

  Write payload to file in yaml format.

  pypyr.steps.filewritejson

  Write payload to file in json format.

  pypyr.steps.filereplace

  Find & replace any arbitrary search strings in a file.

  pypyr.steps.fileformatyaml

  Find & replace substitution {tokens} in a yaml file.

  pypyr.steps.fileformatjson

  Find & replace substitution {tokens} in a json file.

  pypyr.steps.fileformat

  Find & replace substitution {tokens} in any file.

  pypyr.steps.fetchyaml

  Load yaml file into pypyr context.

  pypyr.steps.fetchjson

  Load json file into pypyr context.

• inline code

  #

  Execute inline code as part of your pipeline. Part of the power of pypyr is that you can seamlessly mix inline code with external executables.

  pypyr.steps.pyimport

  Use external libraries in your !py expressions.

  pypyr.steps.py

  Execute inline python code.

  py string - dynamic python expressions

  Evaluate dynamic python expressions in your pipeline steps and task automation sequences.

• json

  #

  Load, format & write json files.

  pypyr.steps.jsonparse

  Parse json string into pypyr context.

  pypyraws.steps.s3fetchjson

  Fetch a json file from s3 into the pypyr context.

  pypyr.parser.jsonfile

  Put json file into context from a cli arg path input.

  pypyr.parser.json

  Put json input string from cli arg into context.

  pypyr.steps.filewritejson

  Write payload to file in json format.

  pypyr.steps.fileformatjson

  Find & replace substitution {tokens} in a json file.

  pypyr.steps.fetchjson

  Load json file into pypyr context.

  jsonify

  Serialize context object to json string

• lookup order

  #

  This is how pypyr resolves pipeline names and the order in which it searches various folders to find the matching pipeline.

  pipeline look-up order

  pypyr looks for pipeline files & custom code modules on the filesystem in these directories.

  pypyr.loaders.file

  The default file loader discovers and loads pipelines from the filesystem.

  pypyr.loaders.string

  Inject pipelines directly as strings.

• loops

  #

  Repeat any step in a loop.

  loops

  How to loop or iterate over a step.

  while loop decorator

  Repeat step until stop condition is True or up to the maximum iteration count.

  retry decorator

  Retry step until it succeeds.

  foreach loop decorator

  Repeat step for each item in list.

• paths

  #

  Work with filesystem paths & globs.

  pypyr.steps.pathcheck

  Check if paths exist on filesystem.

  pypyr.steps.glob

  Get paths from glob expression.

• pipeline format

  #

  The pypyr pipeline format is simple, human-readable yaml.

  pipeline yaml structure

  The format & structure of a pipeline.

  step decorators

  Selectively run, skip, repeat, loop & handle errors on individual steps.

  while loop decorator

  Repeat step until stop condition is True or up to the maximum iteration count.

  skip decorator

  Skip this step if condition True.

  run decorator

  Run this step only if condition True.

  retry decorator

  Retry step until it succeeds.

  swallow decorator

  Swallow step error & continue with pipeline.

  onError decorator

  Add custom error data on step error.

  description decorator

  Descriptive text that prints to output when step runs.

  comment decorator

  Annotations for pipeline authors.

  in decorator

  Add arguments to context for the current step.

  foreach loop decorator

  Repeat step for each item in list.

• print

  #

  Print output to console & log.

  pypyr.steps.echo

  Echo context value of echoMe to the output.

  description decorator

  Descriptive text that prints to output when step runs.

  pypyr.steps.debug

  Pretty print context to output.

• time

  #

  Work with time & timestamps, local & UTC datetime and formatting.

  pypyr.steps.nowutc

  Saves current utc date-time to context nowUtc.

  pypyr.steps.now

  Saves current local date-time to context now.

• toml

  #

  Load, format & write toml files.

  pypyr.steps.filewritetoml

  Write payload to file in toml format.

  pypyr.steps.fetchtoml

  Load toml file into pypyr context.

  pypyr.parser.tomlfile

  Load toml file into context from a cli arg path input.

  pypyr.steps.fileformattoml

  Find & replace substitution {tokens} in a toml file.

• yaml

  #

  Load, format & write yaml files.

  pypyraws.steps.s3fetchyaml

  Fetch a yaml file from s3 into the pypyr context.

  pypyr.parser.yamlfile

  Put yaml file into context from a cli arg path input.

  pypyr.steps.filewriteyaml

  Write payload to file in yaml format.

  pypyr.steps.fileformatyaml

  Find & replace substitution {tokens} in a yaml file.

  pypyr.steps.fetchyaml

  Load yaml file into pypyr context.