Gaussian Process Regression
A Gaussian process places a distribution over functions, giving predictions with calibrated uncertainty that grows away from the data.
A distribution over functions
A Gaussian process (GP) treats an unknown function as a random object: any finite set of its values is jointly Gaussian. It is specified by a mean function, often taken as zero, and a covariance or kernel function that says how correlated the function's values are at two inputs. Nearby inputs are strongly correlated, so the function is smooth; distant inputs are nearly independent.
Prediction by conditioning
Given training data, prediction at a new input is done by conditioning the joint Gaussian on the observed values. This yields both a predictive mean and a predictive variance in closed form. The mean interpolates the data; the variance is small near observations and grows in regions with no data, giving an honest measure of what the model does not know.
The equations
With kernel matrix K on training inputs, noise variance s2, observations y, and cross-covariance k* to a test point, the predictive mean is k*^T (K + s2 I)^-1 y and the predictive variance is k(x*,x*) minus k*^T (K + s2 I)^-1 k*. The single matrix inverse is shared across all predictions.
Strengths
- Built-in, calibrated uncertainty estimates
- Works well with very few data points
- Flexible behavior chosen through the kernel
- Interpretable hyperparameters such as length scale and signal amplitude
The cost
The exact method requires inverting an n-by-n matrix, which scales as the cube of the number of points and becomes impractical beyond a few thousand observations. Sparse and approximate GPs address this. Despite the cost, GPs remain the default choice when data are expensive and honest uncertainty matters, as in the design of physical experiments and surrogate-based optimization.