The Bit and the Nat

The bit and the nat are units of information, differing only by the logarithm base used to measure it.

Information as a unit

Information theory measures how much an outcome reduces uncertainty. The amount is a logarithm of the number of equally likely possibilities. The base of that logarithm names the unit.

The bit

Kronos motion — number counters

Using base 2 gives the bit (short for binary digit in this context, the unit shannon). One bit is the information gained from learning the outcome of a fair coin flip — one of two equally likely results.

The nat

Using the natural logarithm, base e, gives the nat. One nat equals about 1.4427 bits. Nats are natural in mathematics and machine learning because the natural log has a clean derivative.

Converting between them

Since logarithms in different bases differ only by a constant factor, information in nats times log₂(e) gives bits, and bits times ln(2) gives nats. The choice of unit never changes the underlying quantity, only its scale.

Other units

Base 10 gives the hartley or dit, worth about 3.32 bits. Whatever the base, the surprise of an outcome of probability p is the log of 1/p, and expected surprise over all outcomes is the entropy.

python

import math
bits = 3
nats = bits * math.log(2)
print(nats)                       # 2.079...
print(nats / math.log(2))         # back to 3 bits