Distributed Training Strategy

How Kronos splits large model training across many GPUs with data, tensor, and pipeline parallelism without losing reproducibility.

When one GPU is not enough

Some Kronos models are too large or their datasets too big for a single GPU. Distributed training spreads the work across many devices, but naively splitting a model can wreck both efficiency and reproducibility. Kronos chooses a parallelism strategy per model, matched to whether the bottleneck is data volume, model size, or both.

Three axes of parallelism

Data parallelism replicates the model and splits the batch, synchronizing gradients with an all-reduce each step; it suits models that fit on one GPU but have large datasets. Tensor parallelism splits a single layer's math across GPUs for models too big to fit; pipeline parallelism splits the model into stages across devices. Large jobs combine them.

• Data parallelism: split the batch, all-reduce gradients
• Tensor parallelism: split individual layers
• Pipeline parallelism: split the model into staged layers
• Hybrid combinations for the largest models

Communication is the constraint

Every parallelism axis adds communication, so the strategy is chosen with the interconnect in mind. Data-parallel all-reduce scales with model size and GPU count; tensor parallelism demands very fast intra-node links. Kronos places ranks topology-aware so the heaviest traffic stays on the fastest links, keeping strong scaling alive.

Distributed training must remain reproducible. Gradient reductions can be order-sensitive, so runs that feed certified twin models fix the reduction order and seeds, accepting a small performance cost to guarantee that a retrained model can be reproduced. Reproducibility is a constraint on the strategy, not an afterthought.

These jobs are gang-scheduled: all ranks start together or the synchronous training deadlocks. Orchestration allocates the full GPU set and the required fabric topology atomically. The strategy applies equally to breeder and burner models, since both draw from the same GPU forge and the same distributed-training machinery.