Shannon Entropy

Shannon entropy measures the average uncertainty, in bits, of a random source's outcomes.

Definition

For a source with outcomes of probabilities pᵢ, the entropy H is the negative sum of pᵢ log₂ pᵢ. It is the expected number of bits needed to describe one outcome, and it is measured in bits when the log is base 2.

Intuition

Kronos motion — uncertainty

Entropy is high when outcomes are unpredictable and low when one outcome dominates. A fair coin has entropy 1 bit; a two-headed coin has entropy 0 because the result is certain.

Maximum and minimum

For n possible outcomes, entropy is maximized at log₂ n when all outcomes are equally likely, and is 0 when one outcome has probability 1. Any skew in the distribution lowers the entropy.

Why it is the right measure

Shannon showed that entropy is the unique measure (up to scale) satisfying natural axioms: it is continuous, maximal for uniform distributions, and additive for independent sources. It sets the fundamental limit on lossless compression.

Connection to coding

The source coding theorem says the average code length for a source cannot beat its entropy, and can approach it. Entropy is therefore the true information content of a message, independent of any particular encoding.

python

import math
def entropy(ps):
    return -sum(p * math.log2(p) for p in ps if p > 0)
print(entropy([0.5, 0.5]))          # 1.0
print(entropy([0.9, 0.1]))          # 0.469