ML Pipelines

This section covers how we build, configure, and deploy machine learning pipelines using Hydra + hydra-zen for configuration, Flyte for orchestration, and Scaffold as the glue between them.

Contents:

Why these tools?

As ML projects move from proof-of-concept to production, managing configuration, reproducibility, and cloud execution manually becomes prohibitively expensive. Classical software engineering has solved this with version control and CI/CD — but ML systems have additional dimensions: you need to version not just code but also configuration and data, and you need to build and deploy not just a package but an entire training system.

Manual steps compound quickly. Imagine that every change to a learning rate required you to manually rebuild a Docker image, push it, register the workflow, and trigger a run. The maintenance cost would be unsustainable.

The combination of hydra-zen and Flyte solves this:

  • hydra-zen keeps all configuration in Python — versioned, composable, type-checked. No scattered YAML files. Configs are first-class code artifacts that live next to the logic they configure and travel with it through version control.

  • Flyte orchestrates multi-task pipelines in Kubernetes pods, provides caching, and gives you a UI to monitor executions, inspect outputs, and re-run specific tasks. Pipelines and their configuration become versioned, reproducible artifacts that can be delivered to a client or research partner, or replayed exactly months later.

  • Scaffold’s Flyte launcher collapses the five manual steps of building images, serializing, registering, and executing a workflow into one CLI call — from a developer machine or from a CI/CD pipeline.

Together these tools let you reach a state where a single config change in version control triggers CI/CD to test, build, register, and run a new version of the training workflow automatically — what Google refers to as MLOps level 2 and Martin Fowler calls CD4ML.

When should you use Flyte?

Local execution is always the starting point. Reach for Flyte when you need any of the following — and especially when you need several of them together:

  • Scale — a task needs more RAM, GPUs, or parallelism than your laptop can provide

  • Reproducibility — every run is a versioned, inspectable artifact in the Flyte UI

  • Scheduling — workflows that run on a cron schedule without manual intervention

  • Deliverability — pipelines as versioned artifacts, shareable with clients or partners

  • CD4ML — CI/CD automatically registers and runs a new workflow version on every merge

How it works at a glance

A pipeline file follows a five-step structure:

  1. Configure Logic — define your domain classes and create builds() configs for them

  2. Implement Logic — write plain Python functions (no Flyte or hydra dependencies)

  3. Configure Tasks and Workflow — compose task and workflow configs; register named variants

  4. Implement Tasks and Workflow — wrap functions in @runtime_task and @workflow

  5. Run — main() handles local execution; __main__ flushes stores and calls main()

See Quickstart for a minimal working example.