vault / mlops / kubeflow.md

Kubeflow Pipelines

ML platform 74 words 3 outgoing 2 backlinks

Kubeflow Pipelines turns train-test-deploy into a DAG of containerized components on Kubernetes, each with explicit inputs and outputs. It earns its weight when pipelines are long, rerunnable and shared; a two-step nightly job is happier as a script. Gotchas: a component is a container, so pin its image and arguments or “reproducible” is a rumor; caching is input-based, so data that changes silently under the same inputs replays stale results.

Reference: Kubeflow Pipelines docs.

LINKS IN THIS NOTE

[[ Reproducibility ]] Same data, same code, same result - or at least a record of why not. The minimum kit: pinned dependencies, versioned data or a snapshot reference, seeded randomness, and the run's config stored next to its output. "Worked on rerun"... [[ Kubernetes ]] Kubernetes stores desired state while controllers keep trying to make the running system match it. A Deployment manages replicas, a Service gives traffic a stable destination, and Helm packages configuration. That mental model is more useful than memorising object names... [[ Serving SDK ]] A trained model is not yet a service. The shared path should own the repeated work - authentication, tracing, deployment conventions - and stay narrow: a small ordinary path, inspectable behaviour, errors that suggest the next action. Defaults save configuration;...