Vision Transformer (ViT)
The Vision Transformer treats an image as a sequence of patches and applies a standard transformer encoder, showing convolution is not required for strong image models.
From pixels to tokens
The Vision Transformer (ViT) reframes image recognition as a sequence problem. An input image is split into a grid of fixed-size non-overlapping patches, typically 16x16 pixels. Each patch is flattened and projected by a single learned linear layer into a vector of the model dimension. The resulting sequence of patch embeddings is exactly the kind of input a transformer encoder consumes, so the architecture that dominated language modeling transfers to vision with almost no structural change.
Positional information and the class token
A transformer is permutation-invariant, so ViT adds learned position embeddings to the patch embeddings to encode where each patch sat in the grid. A special learnable class token is prepended to the sequence; after the encoder stack, its final hidden state is fed to a small classification head. Every patch attends to every other patch in each layer, giving the model a global receptive field from the first block rather than the gradually widening one a convolutional network builds up.
Encoder blocks
Each block is the familiar pattern: layer normalization, multi-head self-attention, a residual add, then layer normalization, a two-layer MLP with a GELU nonlinearity, and another residual add. Stacking these blocks and scaling width and depth gives the ViT-Base, Large, and Huge variants.
- Global context from layer one, unlike the local kernels of a CNN
- Weak spatial inductive bias, so ViT needs large training sets or strong augmentation and distillation to match CNNs on modest data
- Compute scales quadratically with the number of patches, motivating hierarchical and windowed variants
Why the inductive bias matters
Convolutions bake in translation equivariance and locality for free. ViT does not, so on small datasets it underperforms. Given enough data, that same freedom lets it learn relationships a fixed kernel cannot, and it often transfers well after large-scale pretraining. Hierarchical designs reintroduce locality with shifted local-attention windows while keeping the transformer core.
ViT established that a single architecture, the transformer, can serve as a general sequence backbone across modalities. That unification underpins the multimodal and language systems described in CLIP and decoder-only models.