Laplace Approximation

The Laplace approximation fits a Gaussian to the posterior around a trained network's weights using the loss curvature, adding uncertainty after the fact.

The construction

The Laplace approximation takes a network already trained to a loss minimum (a maximum a posteriori estimate) and approximates the weight posterior as a Gaussian centered there. The covariance is the inverse Hessian of the negative log-posterior: flat directions of the loss give large uncertainty, sharp directions give small uncertainty.

Post-hoc uncertainty

Kronos motion — loss cone

A major appeal is that it requires no change to training: you train normally, then estimate curvature. This makes it a lightweight way to add calibrated uncertainty to an existing deterministic network, in contrast to variational methods that change the training objective.

Approximating the Hessian

Full Hessian is infeasible for large networks
Diagonal or block-diagonal approximations reduce cost
Kronecker-factored (KFAC) approximations balance accuracy and scalability
Last-layer Laplace treats only the final layer as Bayesian, very cheap and often sufficient

Prediction

Predictive uncertainty comes from propagating the Gaussian weight posterior through the network, either by linearizing the network (delta method) or by Monte Carlo sampling weights. The linearized version yields closed-form predictive variance and is common in practice.

Cautions

The Gaussian assumption is local and only valid near the mode; multimodal posteriors are not captured. Curvature estimates can be poorly conditioned and often need a prior-precision term tuned on validation data. Despite these limits, last-layer Laplace is a strong, inexpensive baseline for equipping regression surrogates with uncertainty.