Joint Entropy

Joint entropy measures the combined uncertainty of two random variables considered together.

Definition

For two variables X and Y, the joint entropy H(X,Y) is the negative sum over all pairs of p(x,y) log₂ p(x,y). It is the average bits needed to describe an outcome of the pair at once.

Bounds

Kronos motion — uncertainty

Joint entropy is at least as large as either variable's own entropy and at most their sum: max(H(X),H(Y)) ≤ H(X,Y) ≤ H(X)+H(Y). Describing two things together never needs more than describing them separately.

Independence

When X and Y are independent, knowing one tells nothing about the other, and the joint entropy equals the sum of the individual entropies. Any dependence makes the joint entropy strictly smaller than that sum.

Relation to other quantities

Joint entropy links the family of information measures: H(X,Y) = H(X) + H(Y|X), and mutual information equals H(X)+H(Y)−H(X,Y). It is the anchor from which conditional entropy and mutual information are derived.

Symmetry

Order does not matter: H(X,Y) equals H(Y,X), because the joint distribution is the same object viewed either way. This symmetry mirrors that of mutual information.

python

import math
def joint_entropy(pxy):  # pxy: dict {(x,y): p}
    return -sum(p * math.log2(p) for p in pxy.values() if p > 0)
print(joint_entropy({(0,0):0.25,(0,1):0.25,(1,0):0.25,(1,1):0.25}))  # 2.0