CLIP and Contrastive Multimodal Learning

CLIP trains an image encoder and a text encoder together so that matching image-caption pairs land nearby in a shared embedding space.

Aligning two modalities

CLIP (Contrastive Language-Image Pretraining) learns a joint embedding space for images and text. It uses two encoders, one for images (often a Vision Transformer or a convolutional network) and one for text (a transformer), each producing a single vector. Training pulls the vectors of a true image-caption pair together and pushes apart the vectors of mismatched pairs, using large batches of naturally occurring image-text data from the web rather than curated class labels.

The contrastive objective

Within a batch of N image-text pairs, the model computes an N-by-N matrix of cosine similarities between every image and every text. The correct pairs lie on the diagonal. A symmetric cross-entropy loss treats each row and each column as a classification problem whose correct answer is the diagonal entry, scaled by a learned temperature. Maximizing agreement on the diagonal while suppressing off-diagonal similarity produces the alignment.

import torch, torch.nn.functional as F
img = F.normalize(image_features, dim=-1)   # (N, D)
txt = F.normalize(text_features, dim=-1)    # (N, D)
logits = img @ txt.t() * temperature        # (N, N)
labels = torch.arange(len(img))
loss = 0.5*(F.cross_entropy(logits, labels) + F.cross_entropy(logits.t(), labels))

Zero-shot classification

Because CLIP maps text and images into the same space, it classifies without task-specific training. To recognize a set of categories, each category name is written as a short caption, encoded to a vector, and the image is assigned to whichever caption vector it is closest to. Changing the label set requires no retraining, only new text prompts.

• Robust to distribution shift compared to models trained on fixed label sets
• Prompt wording measurably affects accuracy
• Retrieval works in both directions: find images for a caption or captions for an image
• The image encoder transfers well as a general visual backbone

A foundation for multimodal systems

CLIP showed that natural-language supervision at scale yields flexible, transferable visual features. Its embeddings became a building block for generation and retrieval systems and informed how modalities are combined; see fusion strategies. The contrastive idea also connects to metric learning covered in triplet networks.