Contrastive Learning

Contrastive learning trains representations by pulling similar examples together and pushing dissimilar ones apart.

Learning by comparison

Contrastive learning shapes an embedding space using only the relation between examples: representations of similar items (positive pairs) are pulled close, and representations of dissimilar items (negatives) are pushed apart. Because the supervision comes from these relations, which can be generated automatically, it is a leading form of self-supervised learning.

Building positive pairs

In the self-supervised image setting, two random augmentations of the same image (crop, color shift, blur) form a positive pair; every other image in the batch is a negative. The model learns representations invariant to those augmentations while remaining distinctive across different images. The choice of augmentations strongly determines what invariances the model acquires.

The InfoNCE objective

The dominant loss is InfoNCE: for an anchor, it maximizes the similarity to its positive relative to the similarities to all negatives, formulated as a softmax cross-entropy over similarity scores scaled by a temperature. More and harder negatives generally improve the learned space, which motivated large batches and memory banks of stored embeddings.

# InfoNCE for a batch of L2-normalized embeddings
logits = (z @ z.T) / temperature
logits.fill_diagonal_(float('-inf'))
loss = cross_entropy(logits, positive_index)

Avoiding collapse

Without negatives, a model can cheat by mapping everything to one point (representation collapse). Negatives prevent this, and non-contrastive methods achieve the same end with stop-gradients, predictor networks, or redundancy-reduction objectives instead. Contrastive pretraining also underlies multimodal models that align images and text in a shared space, and it is closely related to metric learning, which pursues the same geometric goal with explicit label supervision.