Data Pipelines: From Sensor to Insight
A data pipeline is the ordered set of stages that move raw measurements into forms that people and models can use reliably.
What a pipeline does
A data pipeline is a chain of stages that carries data from where it is produced to where it is consumed. Each stage transforms, validates, or routes the data, and each has clear inputs and outputs so failures can be located. A well-built pipeline turns a chaotic stream of raw readings into a dependable, queryable record without a human touching every byte.
Pipelines are usually described as a directed acyclic graph (DAG): nodes are processing steps, edges are data dependencies, and the absence of cycles means the work has a well-defined order. Orchestration tools schedule the nodes, retry failed ones, and record when each ran.
Typical stages
- Ingestion: pull or receive raw data from instruments, logs, or upstream systems.
- Validation: check ranges, schemas, and timestamps; quarantine bad records.
- Transformation: unit conversion, calibration, resampling, feature derivation.
- Storage: write to a database, object store, or archive in a durable format.
- Serving: expose the result to dashboards, models, or downstream pipelines.
Batch and streaming
Batch pipelines process bounded chunks on a schedule; streaming pipelines process records continuously as they arrive. Many programs run both: a low-latency stream for live monitoring and a batch pass for the authoritative archive. See batch vs stream processing.
Idempotence and replay
A robust stage is idempotent: running it twice on the same input produces the same output, so a retry after a crash does no harm. Combined with immutable raw storage, this lets an operator replay a pipeline from any point to reproduce a result or backfill a fix.
In a fusion program
For a simulation-and-diagnostics program like Kronos, pipelines carry outputs from plasma physics runs and, in the future, from instrument channels on the Hyperion breeder design. Because current machines are design and simulation rather than built hardware, the dominant pipeline traffic today is simulation output, calibration references, and derived figures for the published record.