Change Data Capture
Change data capture streams every insert, update, and delete from a source database so downstream systems stay synchronized in near real time.
Keeping copies in sync
When a transactional database is the source of truth and analytical or search systems need a fresh copy, something must propagate changes. Polling for changed rows is slow, misses deletes, and burdens the source. Change data capture (CDC) instead streams the exact sequence of changes as they happen, so downstream copies stay synchronized without repeated full extracts.
Log-based capture
The robust way to capture changes is to read the database's own transaction log, the write-ahead log it already maintains for durability. Every committed insert, update, and delete appears there in order. A CDC connector tails this log and emits each change as an event. Because it reads the log rather than querying tables, it captures deletes, imposes little load on the source, and preserves exact ordering.
Why log-based beats query-based
- Captures deletes, which a query for changed rows cannot see
- Adds negligible load compared to repeated scans
- Preserves the true order of changes
- Catches intermediate updates a periodic poll would miss
Snapshot plus stream
A downstream copy needs a starting point. CDC begins with a consistent snapshot of the current table, then switches to streaming changes from the log position captured at snapshot time. Getting the handoff exactly right, so no change is missed or double-applied at the boundary, is the subtle part; connectors solve it by recording the log offset the snapshot corresponds to.
Idempotent application
Because delivery is usually at-least-once, the same change event may arrive twice. Downstream application must be idempotent: applying a change by primary key, so a repeated update produces the same final state. This lets consumers tolerate duplicates and safely resume after a failure by replaying from a known offset. CDC underpins keeping search indexes, embeddings, feature stores, and analytical warehouses current with an operational database. See event streaming, OLAP vs OLTP, and real-time analytics.