Scientific Ml

Bayesian Neural Networks

Bayesian neural networks treat weights as distributions rather than fixed values, yielding predictions with principled uncertainty.

Weights as distributions

A standard neural network learns a single best value for each weight. A Bayesian neural network instead maintains a probability distribution over each weight, expressing uncertainty about the right value. Prediction averages over these distributions, so the output is itself a distribution whose spread reflects the model's confidence.

The Bayesian recipe

One starts with a prior over the weights, updates it with the observed data through Bayes' rule to obtain a posterior, and predicts by integrating over that posterior. The integral is intractable for real networks because the posterior over millions of weights has no closed form, so the practical work is in approximating it.

Approximation methods

Variational inference: fit a simple distribution to the posterior by optimization
Markov chain Monte Carlo: sample weights from the posterior, accurate but costly
Laplace approximation: fit a Gaussian around a trained network's optimum
Monte Carlo dropout: interpret dropout as an approximate posterior

What the uncertainty captures

The spread of a Bayesian network's predictions grows in regions with little data and shrinks where data are dense, capturing epistemic uncertainty, the model's ignorance. Adding a learned noise term captures aleatoric uncertainty, the data's inherent randomness. Together they give a fuller picture than any point estimate.

Costs and alternatives

Bayesian neural networks are more expensive to train and to query than ordinary ones, and the quality of their uncertainty depends heavily on the approximation used. In practice, deep ensembles often match or beat them for uncertainty with less mathematical machinery. Bayesian networks remain valuable when a coherent probabilistic treatment or informative priors are important to the problem.