Data Systems

Data Lineage

Data lineage traces how data flows and transforms from source to destination, answering where a value came from and what would be affected if it changed.

Tracing the flow

Data lineage is the recorded map of how data moves and changes: which sources feed which tables, which transformations apply, and which reports and models consume the result. It turns a tangle of pipelines into a directed graph you can walk in either direction to answer two recurring questions.

Upstream and downstream questions

Kronos motion — what is fusion

Walking upstream answers "where did this number come from?" Given a suspicious figure in a report, lineage traces back through the transformations and joins to the raw sources, so you can find where an error entered. Walking downstream answers "what does this affect?" Given a source that is about to change or was found corrupt, lineage lists every table, dashboard, and model that depends on it, so you can assess impact before acting.

Granularity: table versus column

Table-level lineage: which tables feed which tables
Column-level lineage: which source columns feed which output columns
Column-level is more precise but harder to capture
It answers whether a change to one field actually affects a given output

How lineage is captured

Lineage can be parsed statically from transformation code and query logs, inferring dependencies from the SQL and pipeline definitions, or emitted at runtime by instrumented tools that report what they read and wrote. Parsing covers existing code without changes but struggles with dynamic logic; runtime emission is precise but requires every tool to participate. Most systems combine both.

Lineage as evidence

Beyond debugging and impact analysis, lineage is evidence. For reproducible research and for regulated reporting alike, being able to show the exact path from raw input to published result is what makes the result defensible. When a physics figure is questioned, lineage from the raw simulation output through every transformation to the plotted value is the answer to "how was this produced?" See ML metadata, data catalogs, and content-addressable storage.