UTF-8

UTF-8 is a variable-length encoding of Unicode that stays byte-compatible with ASCII.

Variable length

UTF-8 encodes each code point in one to four bytes. Code points 0–127 use a single byte identical to ASCII, and higher code points use two, three, or four bytes as needed.

How the bytes are marked

Kronos motion — number counters

A leading byte's high bits announce the sequence length: 0xxxxxxx for one byte, 110xxxxx for two, 1110xxxx for three, 11110xxx for four. Continuation bytes always start with 10, so any byte's role is self-evident.

Why the design is clever

ASCII text is valid UTF-8 unchanged
No byte of a multi-byte sequence can be mistaken for an ASCII character
Truncation is detectable because continuation bytes are distinct
Byte-order marks are unnecessary since it is byte-oriented

Self-synchronization

Because continuation bytes are unmistakable, a decoder that starts mid-stream can find the next character boundary by skipping bytes that begin with 10. This resilience makes UTF-8 robust to partial reads.

Dominance

UTF-8 is the default encoding of the web and most modern systems. Its ASCII compatibility eased migration, and its compactness for Latin text made it the practical universal choice.

python

c = '€'  # euro sign U+20AC
b = c.encode('utf-8')
print(b.hex())        # e282ac  (three bytes)
print(b.decode('utf-8') == c)  # True