AdamW Optimizer

AdamW fixes how Adam applies weight decay, decoupling it from the adaptive gradient scaling to make regularization behave as intended.

Adam with corrected weight decay

Adam adapts the step size for each parameter using running estimates of the first and second moments of its gradients. The original Adam implemented weight decay by adding an L2 penalty to the loss, which folds the decay term into the gradient. AdamW showed this is wrong when combined with adaptive scaling: the penalty gets divided by the same per-parameter factor as the gradient, so parameters with large gradient histories are decayed less, distorting the regularization. AdamW instead applies weight decay directly to the weights, decoupled from the adaptive update.

The decoupled update

AdamW performs the ordinary Adam step from the gradient, then separately shrinks each weight toward zero by a factor proportional to the learning rate and the weight-decay coefficient. Because the decay no longer passes through the second-moment normalization, every parameter is regularized by the same relative amount, which is what weight decay is supposed to do. This small change consistently improves generalization for large models.

# per step, after computing Adam's m_hat and v_hat
w = w - lr * m_hat / (v_hat.sqrt() + eps)   # adaptive gradient step
w = w - lr * weight_decay * w               # decoupled decay

• Weight decay is applied to weights, not merged into the gradient
• Regularization strength is consistent across parameters
• Keeps Adam's fast, robust adaptivity and gentle warmup behavior
• The de facto optimizer for transformers and large language models

Practical notes

AdamW's momentum and variance estimates are bias-corrected in early steps so the initial updates are not artificially small. It is usually paired with a learning-rate schedule that warms up then decays, and weight decay is commonly excluded from bias and normalization parameters, which should not be pulled toward zero. Compared with RMSprop, AdamW adds momentum and the corrected decay; compared with Lion, it stores more state per parameter but is well understood and reliable.