Embeddings

An embedding maps discrete items into a continuous vector space where geometric closeness reflects semantic similarity.

What an embedding is

An embedding is a learned mapping from discrete objects, such as words, users, products, or graph nodes, into vectors of real numbers. The mapping is trained so that similar items land near each other in the vector space and dissimilar items land far apart. This converts symbols that a network cannot process directly into a dense, continuous representation it can compute with, while capturing relationships in the geometry of the space.

Why not one-hot

The naive way to represent a discrete item is a one-hot vector: all zeros except a single one at the item's index. One-hot vectors are enormous (one dimension per item), sparse, and carry no notion of similarity, since every pair is equally distant. An embedding replaces this with a compact dense vector, typically tens to a few thousand dimensions, that packs meaning into every component and places related items close together.

How embeddings are learned

An embedding layer is simply a lookup table of vectors, one per item, whose entries are parameters trained by gradient descent along with the rest of the model. During training the vectors move so that they help the task, whether predicting a neighboring word, classifying an image, or scoring a recommendation. The result is a representation shaped by the objective, so embeddings learned for different tasks capture different notions of similarity.

import torch.nn as nn
emb = nn.Embedding(num_embeddings=50000, embedding_dim=256)
# ids -> vectors; the table's weights are learned during training

Geometry of meaning

Distances and directions in embedding space become meaningful. Cosine similarity between two vectors measures relatedness, powering search and recommendation by nearest neighbors. Directions can encode attributes, the classic example being that vector arithmetic on word embeddings approximately yields analogies. This structure lets downstream systems reason about similarity with simple linear-algebra operations.

Where embeddings are used

Embeddings underlie nearly all modern machine learning: word and token embeddings in language models, item and user embeddings in recommenders, node embeddings in graph learning, and joint image-text embeddings in multimodal models. Vector databases store embeddings to enable semantic search and retrieval-augmented generation. In scientific settings, embeddings of materials, molecules, or experimental configurations let models compare and retrieve similar cases across large datasets.

• Maps discrete items to dense continuous vectors.
• Replaces sparse, similarity-free one-hot encoding.
• Learned as a lookup table via gradient descent.
• Geometry encodes similarity, enabling search and retrieval.