Conditional Entropy

Conditional entropy measures the uncertainty remaining in one variable once another is known.

Definition

The conditional entropy H(Y|X) is the average uncertainty of Y given knowledge of X. It equals the joint entropy minus the entropy of X: H(Y|X) = H(X,Y) − H(X).

Intuition

Kronos motion — uncertainty

If knowing X pins down Y completely, then H(Y|X) is 0. If X tells nothing about Y, then H(Y|X) equals H(Y). Conditioning can only reduce or preserve uncertainty, never increase it.

The chain rule

Joint entropy decomposes as H(X,Y) = H(X) + H(Y|X). This chain rule extends to many variables, letting a complex joint distribution be described one variable at a time in terms of what remains uncertain.

Not symmetric

Unlike joint entropy and mutual information, conditional entropy is generally asymmetric: H(Y|X) need not equal H(X|Y). The two differ by the difference in the variables' own entropies.

Use in modeling

Conditional entropy underlies decision trees and feature selection: a feature is informative about a label when conditioning on it sharply lowers the label's entropy. That reduction is exactly mutual information.

python

import math
def cond_entropy(pxy, px):
    h = 0.0
    for (x,y), p in pxy.items():
        if p > 0:
            h -= p * math.log2(p / px[x])
    return h