Model Selection Principles
Choosing among candidate models means balancing fit against complexity and, above all, estimating how each will perform on unseen cases.
The Central Tension
Any model can be made to fit observed data better by adding flexibility, but the goal is usually performance on data not yet seen. Model selection is the disciplined trade-off between goodness of fit, how well a model matches the data at hand, and generalization, how well it will do on new data. Optimizing fit alone reliably selects the wrong model.
Tools for the Trade-off
- Held-out validation and cross-validation: estimate out-of-sample error directly.
- Information criteria: score fit minus a penalty for the number of parameters.
- Regularization paths: let the data decide how much complexity to keep.
- Bayesian model comparison: weigh models by how well they predicted the data on average.
Cross-Validation as the Workhorse
Cross-validation repeatedly trains on part of the data and tests on the held-out remainder, giving a direct estimate of generalization without needing a theoretical penalty. Its main pitfall is leakage: if any choice is tuned using the test folds, the estimate becomes optimistic, and the reported error understates the truth.
Selection Is Not Confirmation
Picking the best of several candidates does not make the winner correct; it makes it the least bad of what was tried. The chosen model still requires validation against reality, and its selection process is itself a source of optimism that should be accounted for. A model chosen from many looks better than it is, simply by selection.
Purpose Governs Choice
The right model depends on the question. A model for prediction, one for mechanistic understanding, and one for extrapolation to new regimes may differ even on the same data. Kronos design work states the purpose a model serves and reports which effects it includes, so its selection can be judged against the use it is put to.