Unicode

Unicode assigns a unique number to every character across the world's writing systems.

One number per character

Unicode gives each character a code point, written U+ followed by a hexadecimal number. It spans over a million possible code points, of which many tens of thousands are assigned, covering scripts, symbols, and emoji.

Code points versus encodings

Kronos motion — number counters

A code point is an abstract number, not a byte layout. Encodings such as UTF-8, UTF-16, and UTF-32 define how to turn code points into bytes. The same text can be stored in any of them and still mean the same thing.

Planes

Code points are grouped into 17 planes of 65536 each. The Basic Multilingual Plane holds most common characters; supplementary planes hold rarer scripts, historic writing, and emoji.

Combining characters and grapheme clusters

Some visible characters are built from several code points, such as a base letter plus a combining accent. What a reader sees as one character, a grapheme cluster, may span multiple code points, which complicates counting and slicing text.

Normalization

Because a character can sometimes be encoded more than one way — precomposed or as base plus combining marks — Unicode defines normalization forms that convert text to a canonical representation so equal strings compare equal.

python

s = 'café'
print([hex(ord(c)) for c in s])
import unicodedata
print(unicodedata.normalize('NFC', s) == s)