A Gaussian-Process Posterior Update
Condition a Gaussian process on three observations and compute the predictive mean and variance at a new input by hand.
Problem
A Gaussian process (GP) defines a distribution over functions through a kernel that encodes smoothness. Given data, the posterior at any test point is Gaussian with a closed-form mean and variance, giving both a prediction and a calibrated uncertainty, which is why GPs are favored for expensive-to-evaluate models.
Kernel and conditioning
Use a squared-exponential kernel k(a,b)=exp(-(a-b)^2/(2 l^2)) with length scale l=1. With training inputs X and targets y, the posterior mean at x* is k*' (K + s^2 I)^-1 y and variance is k(x*,x*) - k*' (K + s^2 I)^-1 k*.
import numpy as np
X=np.array([0.,1.,2.]); y=np.array([0.,0.8,0.9]); s2=1e-4
k=lambda a,b: np.exp(-(a[:,None]-b[None,:])**2/2)
K=k(X,X)+s2*np.eye(3)
xs=np.array([1.5])
ks=k(xs,X)
Ki=np.linalg.inv(K)
mu=ks@Ki@y
var=1.0-(ks@Ki@ks.T)
print('mean',round(float(mu),3),'std',round(float(np.sqrt(var)),3))Result
At x*=1.5, between two observed points, the posterior mean interpolates near 0.9 and the standard deviation is small because the test point sits close to data. Far from any observation the variance rises back toward the prior, so a GP honestly reports greater uncertainty where it has seen nothing. The length scale controls how quickly correlation, and confidence, decays with distance.
- GPs give predictive uncertainty for free, which drives Bayesian optimization and active learning.
- Naive GP cost scales as the cube of the data count, so large datasets need sparse approximations.
- Kronos uses GP surrogates to guide where to spend expensive simulation runs in parameter scans.