BERT and Masked Language Modeling
BERT is a bidirectional transformer encoder pretrained by predicting randomly masked tokens, producing context-rich representations for classification and extraction tasks.
Bidirectional context
BERT (Bidirectional Encoder Representations from Transformers) is a transformer encoder stack that reads an entire input at once, letting every token attend to tokens on both its left and right. This contrasts with autoregressive models, which see only past tokens. Bidirectional context is well suited to understanding tasks such as classification, named-entity recognition, and extractive question answering, where the whole sentence is available and the goal is a representation rather than generation.
The masked language modeling objective
To train a bidirectional model without letting it trivially copy the answer, BERT uses masked language modeling (MLM). About 15 percent of input tokens are selected; of those, most are replaced with a special [MASK] token, some are replaced with a random token, and a few are left unchanged. The model must predict the original token at each selected position from the surrounding context. The random and unchanged cases reduce the mismatch between pretraining, which contains [MASK] tokens, and fine-tuning, which does not.
- [CLS] token at the start yields a pooled representation for sentence-level tasks
- [SEP] token separates sentence pairs for tasks like entailment
- Segment embeddings distinguish the first sentence from the second
- WordPiece subword tokenization keeps the vocabulary bounded
Next sentence prediction and its critics
Original BERT added a next-sentence-prediction task, deciding whether one segment truly followed another. Later work found this objective weak and often harmful, and strong successors dropped it, trained longer on more data, and used dynamic masking that reselects masked positions each epoch.
Fine-tuning and transfer
Pretraining is expensive and done once; adaptation is cheap. For a downstream task, a small head is added on top of the pretrained encoder and the whole model is fine-tuned on labeled data. The [CLS] representation drives sentence classification, while per-token outputs drive tagging and span extraction. See fine-tuning for the mechanics.
BERT demonstrated that self-supervised pretraining on unlabeled text, followed by light supervised adaptation, beats training task-specific models from scratch. That recipe reshaped natural language processing and set the template later scaled up by decoder-only generators.