Data Systems

Schema Evolution

Schema evolution lets a dataset's structure change over time without breaking the readers and writers that depend on it.

Schemas change

A dataset's schema, its set of fields and their types, is never final. New fields appear, old ones are deprecated, types are widened. The problem is that producers and consumers deploy on different schedules, so at any moment old and new code must interoperate over the same data. Schema evolution is the discipline of making changes that do not break either side.

Backward and forward compatibility

Kronos motion — confinement time

Two compatibility directions matter. A change is backward compatible if new code can read data written by old code. It is forward compatible if old code can read data written by new code. Full compatibility is both. Which you need depends on deployment order: if readers upgrade first, you need forward compatibility; if writers upgrade first, backward.

Safe and unsafe changes

Adding an optional field with a default is usually fully compatible
Removing a field breaks readers that still require it
Renaming is a remove plus an add, which breaks both directions
Widening a type (int to long) is often safe; narrowing is not
Changing a field's meaning while keeping its name is the most dangerous change of all

Defaults are the mechanism

Compatibility hinges on defaults. If a new field has a default, old data lacking it reads cleanly under the new schema (backward). If old code ignores unknown fields, it can read new data that added them (forward). Formats designed for evolution, such as Avro and Protocol Buffers, build defaults and unknown-field handling into their resolution rules.

Governing change

Because a bad change can break every consumer at once, mature pipelines put schema changes behind a gate: a schema registry validates a proposed schema against a compatibility rule and rejects incompatible changes before they reach production. This turns schema evolution from a source of outages into a controlled, reviewable process. See Avro and schema registry and data contracts.