Optimization

Gaussian Process Surrogates

A nonparametric model that predicts a function and its uncertainty at every point, the standard surrogate for Bayesian optimization.

A distribution over functions

A Gaussian process (GP) defines a probability distribution over functions such that any finite set of function values is jointly Gaussian. It is specified by a mean function (often zero) and a covariance or kernel function that encodes how correlated the outputs are at nearby inputs. Conditioning on observed data yields a posterior that predicts both a mean and a variance at any new point.

Prediction with uncertainty

Kronos motion — operating point

Given training inputs X with outputs y, the GP posterior at a new point x* gives a predictive mean and variance in closed form using the kernel matrix. The mean interpolates the data; the variance is small near observed points and grows in unexplored regions. This calibrated uncertainty is what makes GPs ideal surrogates for deciding where to sample next.

The kernel

Squared-exponential (RBF): very smooth functions, controlled by a length scale.
Matern: adjustable smoothness, often more realistic for physical responses.
The length scale sets how quickly correlations decay with distance; hyperparameters are fit by maximizing the marginal likelihood.

Fitting hyperparameters

Kernel hyperparameters (length scales, signal variance, noise level) are learned by maximizing the log marginal likelihood, which automatically balances data fit against model complexity. Automatic relevance determination uses a separate length scale per input dimension, revealing which inputs matter most.

Cost and scaling

Exact GP inference requires inverting an n-by-n kernel matrix, costing O(n^3) time and O(n^2) memory, which limits exact GPs to a few thousand points. Sparse GPs using inducing points, and structured kernel approximations, extend them to larger datasets. In high dimensions, more data is needed to fill the space, so GPs are most effective in low to moderate dimension.

python

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import Matern
gp = GaussianProcessRegressor(kernel=Matern(nu=2.5))
gp.fit(X, y); mean, std = gp.predict(Xnew, return_std=True)

Gaussian process surrogates cheaply approximate expensive physics simulations, with uncertainty that guides where the next costly run should go.