Learning GP Hyperparameters
Gaussian process hyperparameters are set by maximizing the marginal likelihood, which balances data fit against model complexity automatically.
What must be tuned
A Gaussian process has a few interpretable hyperparameters, typically a length scale, a signal variance, and an observation noise variance. Their values decide whether the fitted function is smooth or wiggly, confident or cautious. Setting them well is what separates a useful GP from a misleading one.
The marginal likelihood
The principled way to choose hyperparameters is to maximize the marginal likelihood, the probability of the observed data under the GP with those hyperparameters, integrating over all possible functions. This quantity has a built-in Occam balance: it rewards fitting the data but penalizes models flexible enough to fit anything. There is no need for a separate validation set.
The objective
The log marginal likelihood has three parts: a data-fit term y^T (K+s2 I)^-1 y, a complexity penalty equal to the log determinant of (K+s2 I), and a constant. Maximizing it trades these off. Its gradients with respect to the hyperparameters are available in closed form, so gradient-based optimizers converge quickly.
Practical concerns
- The objective can be multimodal, so try several random restarts
- A short length scale plus low noise can overfit; watch for it
- The matrix inverse must be numerically stable; add a small jitter to the diagonal
- Standardize inputs and outputs so default initial values are reasonable
Interpreting the result
Learned hyperparameters carry meaning. A long length scale says the function varies slowly; a large fitted noise says the data are noisy or the model is missing structure. In automatic relevance determination, a separate length scale per input dimension is learned, and very long ones reveal inputs the model finds irrelevant, a built-in form of feature selection.