Streaming Data Systems
Streaming systems process an unbounded sequence of records continuously, producing results with low latency as data arrives.
Data that never stops
A stream is a potentially endless sequence of records that arrive over time. Streaming systems process these records as they come rather than waiting to collect a complete batch. This is the right model for live monitoring, alerting, and any case where the value of data decays quickly with age.
Core concepts
- Producer: a source that emits records onto the stream.
- Consumer: a process that reads and acts on records.
- Broker: a durable middle layer that buffers and routes records.
- Offset: a consumer's position, so it can resume after a restart.
- Windowing: grouping records by time to compute aggregates.
Event time versus processing time
Event time is when something happened; processing time is when the system saw it. Network delays and buffering mean records can arrive out of order or late. Robust streaming frameworks track event time and use watermarks to decide when a time window is complete enough to emit, tolerating bounded lateness.
Delivery guarantees
Systems offer at-most-once (may drop), at-least-once (may duplicate), or exactly-once (no loss, no duplicate) semantics. Exactly-once is the strongest and costliest, usually achieved by combining idempotent writes with committed offsets. Choosing the right guarantee is a trade between correctness needs and throughput.
Windowing patterns
- Tumbling: fixed, non-overlapping intervals.
- Sliding: overlapping intervals that advance by a step.
- Session: windows defined by gaps of inactivity.
In a fusion program
Streaming fits live diagnostic monitoring during a pulse and continuous machine telemetry. A common architecture pairs a stream for immediate dashboards and alerts with a batch pass that writes the authoritative archive. See also message brokers.