Model Selection and the No Free Lunch Theorem
No single algorithm wins everywhere; model selection compares candidates fairly to find the best fit for a given problem.
No free lunch
The no free lunch theorem states that, averaged over all possible problems, every learning algorithm performs the same. There is no universally best model. Any method's advantage comes from assumptions that happen to match the structure of a particular problem. This is why model selection is empirical: you must test candidates on your data.
What to compare
- Simple baselines first: logistic/linear regression, a shallow tree, or majority-class prediction.
- Tree ensembles (random forest, gradient boosting) for tabular data.
- SVM or k-NN for smaller, well-scaled problems.
- Neural networks for images, text, audio, and very large datasets.
How to compare fairly
Evaluate every candidate the same way, with cross-validation on the training data and a single metric aligned to the goal. Tune each model's hyperparameters before comparing, or you compare a tuned model against an untuned one. Keep the test set untouched until the final choice is made.
Beyond raw accuracy
- Interpretability: can you explain a prediction to a stakeholder or regulator.
- Latency and memory at prediction time.
- Robustness to shifts in the data distribution.
- Training time and how easily it retrains on new data.
A pragmatic rule
Start simple, establish a baseline, and add complexity only when it earns its keep on held-out data. A simple model that is understood and maintainable often beats a marginally more accurate one that no one can debug. The no free lunch theorem is a reminder to let evidence, not fashion, pick the model.