Creating Extensible Workflows with Off-Label Use of Python

This talk explores how Python type annotations, coroutines, and dataclasses can shape extensible workflow APIs. It uses the Pants build system to show how task authors can benefit from caching, concurrency, and remote execution.

Syllabus

Intro
About me
What is a workflow?
Workflows show up all over the place
Example: Processing Uploaded Images
Workflow is defined by a task graph
Workflow system design
Workflow for software builds
Design goals
Task API design challenges
Python to the rescue!
Building the rule graph
Static validation
Type annotations provide
Why coroutines?
Coroutines are powerful
Coroutines can express concurrency
Coroutines help avoid side effects
Coroutines provide natural control points for
Summary