Overfitting and Underfitting
Overfitting memorizes training noise and fails on new data; underfitting is too simple to capture real structure.
Two ways to fail
A model overfits when it learns the training data too well, including its noise, and so performs poorly on new data. It underfits when it is too simple to capture the underlying pattern and performs poorly even on the training data. Good generalization lives between these extremes.
How to recognize each
- Overfitting: low training error, high validation error, a large gap between them.
- Underfitting: high training error and high validation error, both poor.
- Good fit: low training error and validation error close behind it.
The gap between training and validation performance is the single most useful diagnostic, and it maps directly onto the bias-variance tradeoff: overfitting is high variance, underfitting is high bias.
Fixing overfitting
- Add training data or augment it.
- Apply regularization (L1, L2, dropout, early stopping).
- Simplify the model: fewer parameters, shallower trees, fewer features.
- Use ensembling or cross-validation to detect it early.
Fixing underfitting
Increase model capacity, add or engineer more informative features, reduce regularization, or train longer. If even a flexible model underfits, the features probably lack signal for the target.
The honest test
Only a held-out test set, untouched during development, reveals true generalization. Tuning against the test set leaks information and reproduces overfitting one level up, so keep it sealed until the end. Watching validation error rise while training error keeps falling is the classic signature of overfitting in progress.