The Reproducibility Crisis in Computation
Many published computational results cannot be reproduced; understanding why points to the practices that prevent it.
Results That Vanish
Across many fields, attempts to reproduce published computational results fail at troubling rates. The reported code cannot be found, the version is unclear, the inputs are incomplete, the environment cannot be recreated, or the described method does not produce the reported numbers. This pattern, part of a broader reproducibility crisis, undermines the checkability that distinguishes science from assertion.
Common Causes
- Code not shared, or shared without the exact version that produced the result.
- Inputs and parameters incompletely reported, requiring guesswork to rerun.
- Environment not captured, so the code fails to build or gives different answers.
- Manual, undocumented steps that no one can reconstruct.
- Selective reporting of the runs that worked, without the full record.
Why Computation Should Be Easy to Reproduce
Computational science has an advantage over experimental science: a calculation is deterministic given the same code, inputs, and environment, so in principle it should reproduce exactly. That it so often fails to is a matter of practice, not principle. The gap between what is achievable and what is achieved is filled by version control, environment capture, provenance, and open sharing, all of which are well understood and available.
The Preventive Practices
The crisis has a constructive reading: it identifies precisely the practices that prevent it. Pin the code with version control and cite the exact version. Report inputs completely. Capture the environment in a container or manifest. Record provenance automatically. Share code and data under clear licenses with persistent identifiers. Provide a runbook that regenerates the result on a clean machine. Each practice closes one of the common failure modes.
A reproducibility discipline built from these practices, such as maintaining cold-reproduction runbooks and DOI-pinned deposits with a stated reproduction tier, is what turns a computational result from an unverifiable claim into a checkable finding. The crisis is, in the end, an argument for the discipline this whole section describes.