Event Streaming
Event streaming platforms store ordered, replayable logs of events partitioned for scale, letting many independent consumers read the same history.
The log as a primitive
An event-streaming platform is built on the append-only log: an ordered, immutable sequence of records. Producers append events to the end; the log never mutates past entries. Unlike a queue that deletes on read, the log retains events for a configured window, so a consumer can replay from any point. This single primitive supports messaging, storage, and stream processing at once.
Topics, partitions, offsets
Events are grouped into topics. Each topic is split into partitions, and each partition is an independent ordered log. A record's position within a partition is its offset. Ordering is guaranteed only within a partition, not across a topic, which is the price of horizontal scaling. The partition a record lands in is chosen by a key, so all events with the same key share a partition and stay ordered relative to each other.
Consumer groups
Consumers in a consumer group divide a topic's partitions among themselves, each reading a disjoint subset, so throughput scales with the number of consumers up to the partition count. Different groups read the same topic independently, each tracking its own offsets. This is how one event stream feeds a real-time dashboard, a training pipeline, and an audit log simultaneously without interference.
Durability through replication
- Each partition is replicated across brokers for fault tolerance
- One replica is the leader; followers copy its log
- A write is acknowledged once enough replicas hold it
- If the leader fails, an in-sync follower is promoted
Replay and reprocessing
Because the log is retained and offsets are explicit, a consumer can reset its offset and reprocess history, for example to rebuild a downstream table after fixing a bug. This replayability is what distinguishes an event log from an ephemeral queue and underpins architectures where the log is the source of truth. See message queues, stream processing, and change data capture.