Surrogate-Based Optimization

Replace an expensive objective with a cheap approximate model, optimize on the surrogate, and validate with occasional true evaluations.

Optimizing the cheap stand-in

When each objective evaluation is a costly simulation or experiment, surrogate-based optimization builds a fast approximate model from a modest set of true evaluations, then optimizes that surrogate instead. Because the surrogate is cheap, it can be searched extensively. The most promising surrogate optima are then checked against the true objective, and the model is refined.

Common surrogates

• Gaussian processes: give calibrated uncertainty, ideal in low dimension.
• Radial basis functions: flexible interpolators that scale better with samples.
• Polynomial response surfaces: simple, interpretable, good locally.
• Neural networks and random forests: scale to high dimension and large data.

The workflow

Sample an initial design using a space-filling plan such as Latin hypercube sampling; fit the surrogate; optimize on the surrogate; evaluate the true objective at the surrogate optimum; add the point and refit. This infill loop concentrates expensive evaluations where they most improve the model and the solution.

Model management and trust

A surrogate is only trustworthy where it has data. Trust-region model management restricts each surrogate optimization to a region where the model is accurate, expanding or shrinking it based on prediction success, much like classical trust-region methods. This prevents chasing spurious surrogate optima in poorly sampled regions.

Relation to Bayesian optimization

Bayesian optimization is surrogate-based optimization with a probabilistic surrogate and an acquisition function that uses uncertainty. Non-Bayesian surrogate methods may simply optimize the surrogate mean plus a diversity criterion. Both aim to minimize the number of expensive true evaluations, the dominant cost in simulation-driven design.

X, y = latin_hypercube(bounds, n0), evaluate(X)
for _ in range(budget):
    s = fit_surrogate(X, y)
    x = optimize(s, bounds)
    X.append(x); y.append(expensive_objective(x))

Surrogate models make large physics-simulation parameter studies feasible by standing in for costly runs during the bulk of the search, a common approach in fusion device design exploration.