Versioning Datasets

Data versioning tracks changes to datasets over time so any past state can be identified, retrieved, and reproduced exactly.

Why data needs versions

Code has version control; data needs it too. Datasets change: new records arrive, calibrations improve, errors are corrected. Without versioning, an analysis cites a dataset that may no longer exist in the form it was run against, and a result becomes irreproducible. Versioning gives every meaningful state of a dataset a stable identity.

Content addressing

The most robust approach identifies data by the hash of its contents. Two files with the same hash are the same bytes; a changed file gets a new hash. This makes a version reference unforgeable and self-verifying: the identifier is a fingerprint of the exact data. See data integrity.

Approaches

• Snapshots: store a full copy at each version; simple, storage-heavy.
• Delta versioning: store only changes between versions; compact, more complex.
• Content-addressed stores: deduplicate identical blocks across versions.
• Table time travel: lakehouse formats keep committed table history.

Immutability of the raw record

A strong discipline is to make raw data immutable: it is written once and never edited. Corrections are new versions or derived products, never overwrites. This guarantees that any past analysis can be re-run against the exact inputs it used, and that the history of understanding is preserved rather than erased.

Pinning in analysis

An analysis should record the specific version identifiers of every dataset it consumes, alongside its code version. Then reproducing the result means checking out that code and pulling those exact data versions. See model versioning for the parallel discipline applied to trained models and derived artifacts. For the Kronos record, deposited datasets are frozen and identified so that a published figure always maps to a fixed input state.