Gaussian-Process Kernels
The kernel encodes every assumption a Gaussian process makes about smoothness, scale, and structure, and choosing it well is the modeling task.
The kernel is the model
A Gaussian process is defined by its mean and its covariance function, or kernel. The kernel says how correlated the function values at two inputs are as a function of their separation, which determines how smooth the function is, how quickly it varies, and what patterns it can express. Choosing the kernel is the central modeling decision in GP regression.
Common kernels
- Squared-exponential (RBF): infinitely smooth functions, one length scale
- Matern: tunable roughness via a parameter nu; nu=3/2 and 5/2 are popular and more realistic than RBF
- Rational quadratic: a mixture of length scales
- Periodic: repeating structure with a fixed period
- Linear: for trends that are affine in the inputs
Length scale and variance
Two hyperparameters recur. The length scale sets how far apart inputs must be before their outputs decorrelate - small length scales allow rapid wiggles, large ones enforce slow variation. The signal variance sets the amplitude of the function's departures from the mean. A separate noise variance models observation error.
Automatic relevance determination
Giving each input its own length scale - automatic relevance determination - lets the GP learn which inputs matter. An input with a very large fitted length scale barely affects the output and is effectively irrelevant. This makes the kernel double as a built-in sensitivity analysis.
Building new kernels
Kernels can be combined: sums capture additive structure, products capture interactions, and combining a periodic with a decaying kernel gives quasi-periodic behavior. Valid kernels must be positive semidefinite, and sums and products of valid kernels stay valid, which makes kernel design a compositional craft.
In practice
For smooth engineering responses, a Matern 5/2 kernel with automatic relevance determination is a robust default - smoother than the physically implausible RBF idealization, flexible enough for most Kronos surrogates of the machines, and informative about which physics inputs drive the response.