Word Embeddings

Word embeddings learn vector representations of words from co-occurrence in text, placing words with similar meanings near one another.

The distributional hypothesis

Word embeddings rest on a linguistic idea: words that appear in similar contexts tend to have similar meanings. By training a model to predict which words occur near which, the resulting vectors capture semantic and syntactic similarity. 'King' and 'queen' end up close because they appear in similar surroundings, even though the model never sees an explicit definition of either.

Word2Vec

Kronos motion — neural operator

Word2Vec, introduced in 2013, learns embeddings from raw text with two simple schemes. Skip-gram predicts the surrounding context words from a center word; CBOW predicts the center word from its context. Training with negative sampling, which contrasts real word-context pairs against random ones, makes the method fast and scalable to billions of words. The learned vectors became a widely used off-the-shelf feature.

GloVe

GloVe takes a different route, factorizing a global word co-occurrence count matrix so that the dot product of two word vectors approximates the logarithm of how often they co-occur. It combines the global statistics of matrix methods with the meaningful vector geometry of prediction methods. In practice Word2Vec and GloVe produce comparable, similarly useful embeddings.

Vector arithmetic

A striking property is that relationships appear as consistent directions. The vector from 'man' to 'woman' is roughly parallel to the vector from 'king' to 'queen', so king - man + woman lands near queen. Similar regularities capture verb tense, country-capital pairs, and comparatives. These analogies made the geometry of embeddings tangible, though they hold only approximately and can encode social biases present in the training text.

python

# analogy via vector arithmetic on trained embeddings
# result = emb['king'] - emb['man'] + emb['woman']
# nearest_neighbor(result) ~ 'queen'

Static versus contextual

Word2Vec and GloVe are static: each word has one fixed vector regardless of context, so 'bank' has a single representation across all senses. Modern models like BERT produce contextual embeddings, where a word's vector depends on the sentence it appears in, resolving ambiguity. Static embeddings remain useful for lightweight applications and as an accessible illustration of how meaning can live in geometry, but contextual representations dominate current systems.

Built on the idea that context reveals meaning.
Word2Vec predicts context; GloVe factorizes co-occurrence.
Relationships appear as consistent vector directions.
Static per-word vectors, unlike contextual embeddings from transformers.