filesystem permalink

pypyr has many ready-made functions to read, write & format files on the filesystem.

You can convert between text, json, yaml & toml formats without you having to write code. This is especially handy when you’re generating configuration or template files on-the-fly that you want to inject into other tools that you are automating with pypyr.

cross-platform paths permalink

Windows users, you can specify paths with / or \ as directory separator. pypyr will resolve either correctly for you. Pretty much all platforms except MS Windows use /.

When you’re working with files and you want to have a cross-platform compatible pipeline and you’re using relative paths, front-slash / is the way to go.

encoding permalink

pypyr will use the platform’s default encoding to deal with text-based files. This is utf-8 for most sensible systems, but be aware on Windows it’s still cp1252.

steps permalink

All steps that work with files have an encoding input where you can override the default and explicitly set the encoding:

- name: pypyr.steps.filewrite
      path: out/utf8-example.txt
      payload: "€ ∮ E⋅da = Q,  n → ∞, ∑ f(i) = ∏ g(i), ∀x∈ℝ: ⌈x⌉ = −⌊−x⌋, α ∧ ¬β = ¬(¬α ∨ β)"
      encoding: utf-8

See here for a list of available encodings.

You cannot set encoding in binary mode.

in-out steps permalink

All steps with an in and out input allow you to override the default encoding with three possible inputs:

  • encoding - set the encoding for both in + out.
  • encodingIn - set the encoding for the in file.
  • encodingOut - set the encoding for the out file.

These are all the “format” and “replace” steps.

You can convert files between different encodings by setting a different encoding for the input than the output.

If you don’t specify encodingIn pypyr will default to the value of encoding for the input file.

If you don’t specify encodingOut pypyr will default to the value of encoding for the output file.

If you did not specify encoding either, pypyr will use the default encoding.

context parsers permalink

The file-based context parsers always use the default encoding when reading input files. This is the case when you:

A toml file must always be in utf-8, per the TOML Spec.

default encoding permalink

The platform default is utf-8 for most systems, but be aware on Windows it’s still cp1252. To check your platform’s default encoding, do:

import locale

If you’re getting annoyed with having to change the encoding on each step, you can change the default if you either:

  • Set pypyr’s default encoding in config.
    • This changes the encoding just for pypyr.
  • Set Python to run in utf-8 mode.
    • See PEP 0540.
    • This sets the default encoding to utf-8 for your entire Python environment.
  • Set Python encoding to something other than utf-8.
    • Maybe don’t do this.
    • Check the Python docs if you want to go this route.

If you explicitly set encoding on any filesystem step it will override the default for that particular step.

Windows users, it might be an idea to set your over-all Python runtime to use utf-8 like all other modern platforms do. Then you wouldn’t need to worry about any of this.

See PEP 0540.

TLDR; Set the environment variable PYTHONUTF8 to 1.

filesystem operations permalink

fetchjsonLoad json file into pypyr context.
fetchtomlLoad toml file into pypyr context.
fetchyamlLoad yaml file into pypyr context.
fileformatFind & replace substitution {tokens} in any file.
fileformatjsonFind & replace substitution {tokens} in a json file.
fileformattomlFind & replace substitution {tokens} in a toml file.
fileformatyamlFind & replace substitution {tokens} in a yaml file.
filereadRead file into pypyr context.
filereplaceFind & replace any arbitrary search strings in a file.
filewriteWrite payload to file.
filewritejsonWrite payload to file in json format.
filewritetomlWrite payload to file in toml format.
filewriteyamlWrite payload to file in yaml format.
globGet paths from glob expression.
jsonfilePut json file into context from a cli arg path input.
pathcheckCheck if paths exist on filesystem.
tomlfileLoad toml file into context from a cli arg path input.
yamlfilePut yaml file into context from a cli arg path input.
last updated on .