Data Systems

ML Metadata and Run Lineage

ML metadata is the structured record connecting datasets, executions, and artifacts into a queryable graph of how every model came to be.

Metadata as a graph

Beyond logging individual runs, mature machine-learning systems maintain a metadata store: a graph whose nodes are artifacts (datasets, models, evaluation reports) and executions (training jobs, preprocessing steps), and whose edges record which execution consumed and produced which artifacts. This graph answers lineage questions across the whole lifecycle, not just within one run.

Three node types

Kronos motion — pid vs model

Artifacts: concrete objects with a URI and a content hash
Executions: runs of a component, with their parameters and timing
Contexts: groupings such as a pipeline run or a project, tying nodes together

Why lineage matters operationally

When a production model misbehaves, the first question is "what went into it?" A metadata graph answers it by walking backward: this model came from this training execution, which consumed this dataset artifact, which was produced by this preprocessing execution from these raw inputs. The same walk forward answers "if this dataset is corrupt, which deployed models are affected?" That impact analysis is impossible without recorded lineage.

Caching and skip logic

A metadata store also enables pipeline caching. If a component is asked to run with inputs and parameters identical to a previous execution, the orchestrator can return the prior output artifact instead of recomputing it. Correctness depends on the metadata faithfully capturing every input that affects the output; a hidden dependency not recorded in metadata breaks the cache silently.

Distinguishing tracking from metadata

Experiment tracking is often researcher-facing and run-centric: compare these hundred runs. A metadata store is often system-facing and artifact-centric: trace this deployed artifact to its origins. In practice the two overlap, and modern platforms expose both views over one underlying store. The discipline that makes either trustworthy is recording provenance at the moment of execution rather than reconstructing it later.