run a pipeline from the api
You can run a pypyr automation pipeline programmatically from your own code using the python api.
# ./pipeline-dir/my-pipe.yaml
context_parser: pypyr.parser.keyvaluepairs
steps:
- name: pypyr.steps.echo
in:
echoMe: piper {arbkey} that {anotherkey} again
- name: pypyr.steps.contextsetf
in:
contextSetf:
myoutput: I was set in the pipeline!
input_values:
- '{arbkey}'
- '{anotherkey}'
some_nesting:
down_level:
arb_number: 123
arb_bool: False
Run this pipeline from the cli like this:
$ pypyr pipeline-dir/my-pipe arbkey=pipe anotherkey=song
piper pipe that song again
$
You can run this same pipeline programmatically like this:
# Optional. If you want to see pypyr stdout output, log level should be <= 25
# import logging
# logging.basicConfig(level=25)
from pypyr import pipelinerunner
# You can run via API like this:
pipelinerunner.main(pipeline_name='pipeline-dir/my-pipe',
pipeline_context_input=['arbkey=pipe', 'anotherkey=song'])
# Or like this:
context_out = pipelinerunner.main_with_context(
pipeline_name='pipeline-dir/my-pipe',
dict_in={'arbkey': 'pipe',
'anotherkey': 'song'})
# context_out behaves like a dict
print(context_out['myoutput']) # I was set in the pipeline!
print(context_out['input_values'][0]) # pipe
print(context_out['some_nesting']['down_level']['arb_number']) # 123
Use the pypyr.pipelinerunner
module. There are two sensible entry-points for
most developers:
- main()
- Use this if you want to pass context arguments to the pipeline’s
context_parser
in exactly the same way as the cli does.
- Use this if you want to pass context arguments to the pipeline’s
- main_with_context()
- Use this if you have your own dict-like structure you want to pass to your pipeline.
- This will bypass the pipeline’s
context_parser
and use your input dict directly. - This function also returns the pypyr
Context
object to you after the pipeline completes.
Call whichever you prefer once per pypyr pipeline invocation. Both entry-points are identical for actual pipeline control-of-flow execution - the only difference is how you initialize the context that the pipeline will use.
main entry-point
Use main()
if you want to run a pipeline exactly like the cli does, by using
the pipeline’s context_parser
to initialize context with a list of string
arguments.
pipeline_context_input
takes a list of strings. This is a POSIX style argument
split on the input arguments following the pipeline name as passed from the cli.
$ pypyr mypipeline arg1 "arg 2" arg3=4
Equates to a pipeline_context_input
of:
['arg1', 'arg 2', 'arg3=4']
If you have a single string representing all the input arguments, a convenient way to split it into a list while honoring escape sequences & quotes is to use the built-in python shlex split function.
import shlex
from pypyr import pipelinerunner
# For a pipeline that you'd run from the cli like this:
# $ pypyr pipeline-dir/my-pipe arbkey=pipe anotherkey="song with a space"
in_args = shlex.split('arbkey=pipe anotherkey="song with a space "')
# in_args == ['arbkey=pipe', 'anotherkey=song with a space ']
pipelinerunner.main(pipeline_name='pipeline-dir/my-pipe',
pipeline_context_input=in_args)
input args
import pypyr.pipelinerunner
pypyr.pipelinerunner.main(pipeline_name='arb-pipe',
pipeline_context_input=['arb', 'context input'],
working_dir='arb/dir',
groups=['group1', 'group2'],
success_group='success_group',
failure_group='failure_group',
loader='mypackage.myloader')
pipeline_name
: string. Required.- Name of pipeline, sans .yaml at end.
{pipeline_name}.yaml
is relative to theworking_dir
.
pipeline_context_input
: list of string. Optional.- Initialize the pypyr context with this list of strings.
- Use the python shlex split function on a string to get a parsed list.
- If not specified pypyr will create an empty
Context
object for you, depending on how the pipeline’scontext_parser
handlesNone
input.
working_dir
: Path-like. Optional.- Look for pipelines and modules in this directory.
- Default to the current directory if not specified.
groups
: list of string. Optional.- Step-group names to run in pipeline.
- Default to
steps
if not specified.
success_group
: string. Optional.- Step-group name to run on success completion.
- Default to
on_success
if not specified.
failure_group
: string. Optional.- Step-group name to run on pipeline failure.
- Default to
on_failure
if not specified.
loader
: string. Optional.- Absolute name of pipeline loader module.
- If not specified will use
pypyr.pypeloaders.fileloader
- the standard pypyr pipeline loader.
returns
None. If you want to interact with the Context
object after the pipeline
run completes, call main_with_context
instead.
main with context entry-point
Use main_with_context
instead of main
if you have a a dict-like object you
want to use to initialize the context, rather than using the context parser with
a string input that pypyr needs to parse first.
main_with_context
also returns the context after the pipeline completes,
giving you access to the values the pipeline stored to context during its run.
If you do want to run the pipeline’s context_parser
, use main()
instead.
input args
import pypyr.pipelinerunner
context_out = pypyr.pipelinerunner.main_with_context(
pipeline_name='mypath/arb-pipe',
dict_in={'key': 'value', 'arbkey': 'arb value'},
working_dir='arb/dir',
groups=['group1', 'group2'],
success_group='success_group',
failure_group='failure_group',
loader='mypackage.myloader')
The input arguments are the same as for main()
, with the only difference being
that instead of passing a list of string in pipeline_context_in
you pass a
dict-like object in dict_in
.
pipeline_name
: string. Required.- Name of pipeline, sans .yaml at end.
{pipeline_name}.yaml
is relative to theworking_dir
.
dict_in
: dict. Optional.- Initialize the pypyr
Context
object with this dict. - If not specified pypyr will create an empty
Context
for you. - The function returns the created
Context
object with any mutations made by the pipeline.
- Initialize the pypyr
working_dir
: Path-like. Optional.- Look for pipelines and modules in this directory.
- Default to the current directory if not specified.
groups
: list of string. Optional.- Step-group names to run in pipeline.
- Default to
steps
if not specified.
success_group
: string. Optional.- Step-group name to run on success completion.
- Default to
on_success
if not specified.
failure_group
: string. Optional.- Step-group name to run on pipeline failure.
- Default to
on_failure
if not specified.
loader
: string. Optional.- Absolute name of pipeline loader module.
- If not specified will use
pypyr.pypeloaders.fileloader
- the standard pypyr pipeline loader.
returns
The pypyr context as it is after the pipeline completes. This is of type
pypyr.context.Context()
. The Context
object behaves pretty much like a
standard dict
.
invoke pipeline from api with no input context
Setting an input context is optional.
# ./no-input-context.yaml
steps:
- name: pypyr.steps.py
in:
py: print('hello hello!')
- name: pypyr.steps.contextsetf
in:
contextSetf:
arbkey: I was set in the pipeline!
You can run this pipeline like this:
# For a pipeline ./no-input-context.yaml
# that you'd run from the cli like this:
# $ pypyr no-input-context
from pypyr import pipelinerunner
pipelinerunner.main(pipeline_name='no-input-context')
# or if you want to work with return context from pipeline
out = pipelinerunner.main_with_context(pipeline_name='no-input-context')
print(out['arbkey']) # I was set in the pipeline!
logging
By default python runs with a log level of 30 (WARNING). This means you won’t see pypyr NOTIFY output like pypyr.steps.echo or description output when you invoke the pypyr api from your own code without setting the log-level. This is because as an API pypyr shouldn’t clutter your stdout unless you explicitly tell it to do so.
If you’re invoking pypyr via the API from your own application, it’s your
responsibility to set up and configure logging. If you just want the same log
handlers & formatters that the pypyr cli uses, you can call
pypyr.log.logger.set_root_logger() before
invoking pipelinerunner.main()
or pipelinerunner.main_with_context()
.
import pypyr.log.logger
from pypyr import pipelinerunner
# use the same log format & level defaults as the cli
pypyr.log.logger.set_root_logger()
# For a pipeline that you'd run from the cli like this:
# $ pypyr pipeline-dir/my-pipe arbkey=pipe anotherkey=song
context_out = pipelinerunner.main_with_context(
pipeline_name='pipeline-dir/my-pipe',
dict_in={'arbkey': 'pipe',
'anotherkey': 'song'})
Be aware that pypyr adds a NOTIFY
- 25
custom log-level and a notify()
function to logging
.
log levels
Log level enumeration:
- < 10 gives full traceback on errors.
- 10=DEBUG
- 20=INFO
- 25=NOTIFY (default)
- 30=WARNING
- 40=ERROR
- 50=CRITICAL
set_root_logger input args
Use set_root_logger
when you want to use the same logging format as the cli.
This is optional. Your application is free to define its own log level and
handlers - in which case, don’t bother with set_root_logger()
.
import pypyr.log.logger
pypyr.log.logger.set_root_logger(log_level=25,
log_path='./dir/my-file.log')
log_level
: int. Optional.- Defaults to
25
if not specified.
- Defaults to
log_path
: Path-like. Optional.- If specified, append pypyr output to this file.
- Defaults to
None
if not specified.