Data Quality and Validation
Data quality is measured across dimensions like accuracy and completeness; validation enforces them by testing data as it flows.
Quality is not automatic
Data does not arrive clean. Sensors glitch, timestamps skew, fields go missing, and units get confused. Data quality is the discipline of defining what good means for a dataset and enforcing it, so that downstream analysis is not built on quietly broken inputs. Validation is the machinery that performs the enforcement.
Dimensions of quality
- Accuracy: values reflect the real quantity within known uncertainty.
- Completeness: expected records and fields are present.
- Consistency: values agree across related fields and datasets.
- Timeliness: data is current enough for its use.
- Validity: values obey type, range, and format rules.
- Uniqueness: no unintended duplicates.
Validation in the pipeline
Validation runs as a stage that checks each batch or record against rules derived from the schema and domain knowledge: types, ranges, required fields, referential links, and statistical expectations. Records that fail are quarantined rather than silently dropped or passed, so a human can inspect the cause.
Detecting anomalies
Beyond fixed rules, statistical checks catch subtler problems: a channel whose noise floor jumps, a distribution that shifts, or a value far outside its historical range. These do not always mean bad data, sometimes the physics changed, so anomalies are flagged for review rather than automatically rejected.
Failing loudly
The core principle is that bad data should cause an early, visible failure, not a plausible-looking wrong answer downstream. A validation gate that halts a pipeline is inconvenient; a corrupted result that reaches a published figure is far worse. For a fusion program, validation guards the path from diagnostics and simulation outputs to the analyzed and deposited record.