Kullback-Leibler Divergence

The Kullback-Leibler divergence measures how much one probability distribution differs from a reference distribution.

Definition

The Kullback-Leibler (KL) divergence from a reference distribution P to an approximation Q is D(P ‖ Q) = Σ p(x) log(p(x)/q(x)). It is the expected extra number of bits (or nats) needed to encode data from P using a code built for Q instead of the true P.

Key properties

Kronos motion — training from sim

Non-negative: D(P ‖ Q) ≥ 0, with equality only when P = Q (Gibbs' inequality).
Asymmetric: D(P ‖ Q) ≠ D(Q ‖ P) in general, so it is not a distance metric.
Undefined when Q assigns zero probability to an outcome P allows.

Relationship to entropy

KL divergence decomposes as D(P ‖ Q) = cross-entropy(P, Q) − H(P). Minimizing cross-entropy over Q, with P fixed as the data distribution, is therefore the same as minimizing KL divergence — which is exactly the loss function used to train probabilistic classifiers.

python

import math
def kl(p, q):
    return sum(pi*math.log2(pi/qi) for pi,qi in zip(p,q) if pi>0)
print(round(kl([0.5,0.5],[0.9,0.1]),4))  # 0.7370 bits

Why the asymmetry matters

Minimizing D(P ‖ Q) versus D(Q ‖ P) yields different approximations. The forward direction spreads Q to cover all of P's support (mean-seeking); the reverse lets Q concentrate on one mode (mode-seeking). Variational inference uses the reverse direction, which is why its approximations can underestimate uncertainty.

Where it appears

KL divergence is the objective behind maximum likelihood (minimizing divergence between data and model), variational inference, and model selection criteria. It is the standard way to quantify how well one distribution stands in for another.