The Transformer

The transformer builds sequence models entirely from attention and feedforward layers, enabling the parallel training that produced modern foundation models.

Attention is all you need

The 2017 transformer removed recurrence and convolution from sequence modeling, relying only on attention and position-wise feedforward layers. This eliminated the sequential bottleneck of RNNs: a transformer processes all positions of a sequence in parallel during training, so it scales efficiently to enormous datasets and model sizes. That scalability, more than any single trick, is why the transformer became the dominant architecture.

The block

Kronos motion — training from sim

A transformer layer stacks two sublayers. First, multi-head self-attention lets every position gather context from the whole sequence. Second, a position-wise feedforward network (two linear layers with a nonlinearity) transforms each position independently. Each sublayer is wrapped with a residual connection and layer normalization, which keeps gradients flowing through deep stacks. Dozens of such blocks are stacked.

Encoder, decoder, or both

Encoder-only (BERT): bidirectional attention for understanding tasks like classification and retrieval.
Decoder-only (GPT): causal attention for autoregressive generation; the shape of most large language models.
Encoder-decoder (T5, translation): an encoder reads the source and a decoder generates the target using cross-attention.

Position information

Because attention is permutation-invariant, treating a sequence as an unordered set, the transformer must be told the order of tokens. Positional encodings, added to token embeddings, inject this information. Variants include fixed sinusoidal encodings, learned position embeddings, and rotary encodings that act inside the attention scores.

python

# one transformer block (pre-norm variant)
def block(x):
    x = x + mha(layernorm(x))       # self-attention sublayer
    x = x + ffn(layernorm(x))       # feedforward sublayer
    return x

Scaling and reach

Transformer performance improves predictably as parameters, data, and compute grow, a regularity that motivated ever larger foundation models. The architecture also spread beyond text: vision transformers treat image patches as tokens, and transformers now handle audio, protein sequences, and scientific time-series. The same block, restacked and rescaled, addresses a remarkable range of problems. In research settings, transformer surrogates help interpolate across large simulation datasets for design exploration.

Built only from attention and feedforward layers.
Fully parallel training, unlike recurrent models.
Residuals and layer norm enable deep stacks.
Encoder, decoder, or both, plus positional information.