Optimization

Adam

Combine momentum and per-parameter adaptive scaling with bias correction: the default optimizer for training deep neural networks.

Adaptive moment estimation

Adam maintains two exponentially decaying averages: a first moment m (the mean of gradients, providing momentum) and a second moment v (the mean of squared gradients, providing per-parameter scaling). It combines the momentum of the heavy-ball method with the adaptive step sizes of RMSProp.

The update

Kronos motion — confinement scaling

m_k = b1 * m_{k-1} + (1-b1) * g_k; v_k = b2 * v_{k-1} + (1-b2) * g_k^2. Because m and v start at zero they are biased toward zero early on, so Adam applies bias correction: m_hat = m_k / (1 - b1^k), v_hat = v_k / (1 - b2^k). The update is x_{k+1} = x_k - a * m_hat / (sqrt(v_hat) + eps).

Default hyperparameters

b1 = 0.9 (first-moment decay).
b2 = 0.999 (second-moment decay).
a = 0.001 learning rate; eps = 1e-8.
These defaults work across a wide range of problems with little tuning.

Why it is popular

Adam is robust to gradient scale, converges quickly in early training, and needs little manual tuning, which makes it the default for many deep learning tasks. It handles noisy, sparse gradients and non-stationary objectives well.

Caveats

Adam can converge to worse-generalizing solutions than well-tuned SGD with momentum on some vision tasks, and its adaptive scaling interacts poorly with standard L2 regularization. AdamW decouples weight decay to fix this. Convergence issues on some convex problems led to variants like AMSGrad.

python

m=v=0
for k in range(1, iters+1):
    g = grad(x)
    m = 0.9*m + 0.1*g
    v = 0.999*v + 0.001*g*g
    mh = m/(1-0.9**k); vh = v/(1-0.999**k)
    x = x - lr*mh/(vh**0.5 + 1e-8)

Adam and AdamW train the large surrogate and generative models that support scientific data analysis and engineering design exploration.