GPT and Autoregressive Models

Decoder-only transformers predict the next token given all previous ones, a simple objective that scales into general-purpose language models.

Next-token prediction

An autoregressive language model factorizes the probability of a sequence into a product of conditional probabilities: the chance of each token given all tokens before it. Training maximizes the likelihood of real text under this factorization, which reduces to predicting the next token at every position. Despite its simplicity, this objective forces the model to learn grammar, facts, reasoning patterns, and style, because all of these help predict what comes next.

Decoder-only architecture

Kronos motion — purpose

GPT-style models use a stack of transformer decoder blocks with causal self-attention: a mask prevents each position from attending to future tokens, so the prediction at each step depends only on the past. There is no separate encoder. The same stack that is trained on prediction is used for generation, making the architecture uniform and easy to scale.

python

# causal mask blocks attention to future positions
import numpy as np
def causal_mask(n):
    m = np.triu(np.ones((n, n)), k=1)   # 1 above diagonal
    return np.where(m == 1, -1e9, 0.0)  # added to attention scores

Generation and sampling

To generate text the model produces a probability distribution over the next token, a token is chosen, appended, and the process repeats. Choice strategies trade off diversity and coherence: greedy decoding takes the most likely token, temperature sampling injects randomness, and top-k or nucleus sampling restrict choices to the most probable candidates. These knobs shape how creative or deterministic the output is.

Scaling and emergent behavior

Autoregressive transformers improve smoothly and predictably as parameters, data, and compute increase, described by empirical scaling laws. As models grow, some capabilities appear that smaller models lack, such as in-context learning: performing a new task from examples in the prompt with no weight updates. This behavior made large decoder-only models general-purpose tools rather than single-task systems.

Alignment and adaptation

A raw pretrained model predicts likely text, not necessarily helpful or accurate answers. Instruction tuning and reinforcement learning from human feedback adapt it to follow instructions and behave as an assistant. The same base model can also be specialized by fine-tuning or prompting for domains such as code, scientific writing, or data analysis without retraining from scratch.

Predicts the next token from all previous tokens.
Decoder-only stack with causal masking.
Sampling strategies tune diversity versus coherence.
Scales predictably and shows in-context learning.