Derivative-Free Optimization
Derivative-free optimization minimizes functions using only their values, essential when gradients are unavailable, unreliable, noisy, or expensive.
When gradients are out of reach
Gradient-based methods are the fastest optimizers when derivatives are available and accurate. But many real objectives are black boxes: the output of a legacy simulation, a physical experiment, or a process with discontinuities and noise, where no gradient can be computed or where a numerical gradient would be swamped by noise. Derivative-free optimization (also called black-box or zeroth-order optimization) works entirely from function evaluations.
Families of methods
Derivative-free methods fall into a few groups. Direct-search methods (Nelder-Mead, pattern search, Powell) move a set of sample points by geometric rules. Model-based methods (trust-region derivative-free, Bayesian optimization) fit a surrogate model to sampled values and optimize the surrogate. Stochastic and evolutionary methods (CMA-ES, genetic algorithms, simulated annealing) sample randomly and evolve a population or distribution toward better regions.
- Direct search: geometric moves on sample points, no model
- Model-based: fit a surrogate (quadratic or Gaussian process), optimize it
- Evolutionary: sample and select over generations
- Finite differences: estimate a gradient, fragile under noise
The cost of no gradients
The fundamental limitation is efficiency. Without gradient information, the number of evaluations needed grows quickly with dimension, so derivative-free methods are practical mainly up to tens or low hundreds of variables. When each evaluation is expensive, as in a costly physics simulation, sample-efficient model-based methods such as Bayesian optimization are preferred because they extract maximum information from each evaluation.
Choosing a method
Match the method to the problem. For a smooth, cheap, low-dimensional objective, Powell or Nelder-Mead is a fast first try. For noisy or multimodal problems, CMA-ES is robust. For very expensive evaluations, Bayesian optimization minimizes the number of runs. In engineering design, derivative-free optimization is standard for calibrating and tuning parameters of complex multiphysics simulations whose internal gradients are not exposed.