Learning-Rate Schedules
A learning-rate schedule changes the step size over training, typically warming up then decaying, to balance fast early progress against stable final convergence.
Why a fixed rate is not enough
The learning rate is the single most influential hyperparameter in training. Too large and training diverges or oscillates; too small and it crawls. A fixed value forces one compromise for the whole run, but the ideal step size changes: early on the model benefits from large steps to move quickly, while late in training small steps help settle into a good minimum. A schedule varies the rate over time to match these phases.
Warmup
Many modern models begin with a warmup, in which the learning rate rises linearly from near zero to its peak over the first few hundred or thousand steps. At initialization the gradient statistics used by adaptive optimizers are unreliable and the model is fragile, so a large step can destabilize it. Warmup lets those statistics settle and the weights find a reasonable region before full-size steps are taken. It is essential for large transformers and large-batch training.
Decay shapes
- Step decay: cut the rate by a factor at fixed milestones
- Cosine decay: smoothly follow half a cosine curve down to a small floor
- Linear decay: decrease linearly to zero over the run
- Exponential decay: multiply by a constant factor each step
- Inverse square root: decay proportional to one over the square root of the step
Cosine decay after warmup is a common default because its smooth descent spends time at both high and low rates without abrupt jumps. The related warmup-plus-decay pattern is detailed in warmup and cosine decay.
Practical guidance
The peak learning rate, the warmup length, and the decay shape interact and are usually tuned together. Cyclical schedules that raise and lower the rate repeatedly can help escape poor regions, and restarts can improve some tasks. For fine-tuning a pretrained model, a small peak rate with a short warmup and linear decay is standard, as noted in fine-tuning. The schedule is chosen alongside the optimizer, since methods like AdamW and Lion respond differently to the same curve.