Epoch, Batch, Iteration
The three units that describe how much data a training run has processed.
Definition
In training, a batch is the group of examples used to compute one gradient update, an iteration is one such update, and an epoch is one full pass through the entire training dataset.
On very large datasets, models are sometimes trained for less than a full epoch, seeing each example at most once, because fresh data generalizes better than repeated passes. The right regime depends on whether data or compute is the binding constraint.
The relationship among these units, together with dataset size, determines the total compute a training run consumes, which is the quantity that scaling studies actually vary. Reporting them precisely lets others reproduce a result and reason about its cost. In frontier training, where data may be seen only once, the epoch loses meaning and total tokens processed becomes the natural unit instead.
If a dataset has N examples and the batch size is B, one epoch contains N/B iterations.
Trade-offs
- Larger batches give smoother gradients and better hardware use.
- Smaller batches add noise that can aid generalization.
- More epochs risk overfitting; early stopping guards against it.
Why it matters
These units define the training schedule and interact with the learning rate and hardware. Reporting them makes a training run reproducible, and tuning batch size is a common lever for speed and stability in SGD.
Fusion connection
Batch size and epoch count are logged for every Kronos surrogate so training is reproducible, matching the program's emphasis on verifiable methods.