Transformer
A sequence architecture built on attention that processes all positions in parallel and scales to very large models.
Definition
The transformer replaces recurrence with self-attention, letting every position in a sequence directly attend to every other position. This removes the sequential bottleneck of RNNs and enables massive parallel training.
Self-attention has quadratic cost in sequence length, since every position attends to every other, which motivates a large body of work on efficient approximations for long contexts. Managing this cost is central to extending models to long documents and high-resolution inputs.
The architecture's uniformity is part of its power: the same block, stacked and scaled, works across text, images, audio, and code, which has concentrated research and hardware optimization on one design. This generality, combined with predictable scaling, is why the transformer underpins nearly all frontier models, even as work continues on reducing its quadratic attention cost for very long inputs.
Stacked attention and feed-forward layers, with positional encodings to convey order, form the backbone of modern language and multimodal models.
Why it works
- Parallelism: all positions processed at once.
- Long-range context: direct connections between distant tokens.
- Scalability: performance improves predictably with size and data.
Why it matters
Transformers are the architecture behind large language models and much of modern AI. Their ability to scale efficiently on parallel hardware is central to recent progress.
Fusion connection
Attention-based models handle multivariate plasma sequences with long-range structure, and their parallel training suits the GPU clusters Kronos uses for surrogate development.