Lambda and Kappa Architectures
Lambda and kappa are two blueprints for combining historical accuracy with real-time freshness in one analytical system.
The reconciliation problem
A system that must serve both accurate historical results and fresh real-time results faces a tension: batch processing is accurate but slow, stream processing is fast but approximate and stateful. Lambda and kappa architectures are two answers to how these should coexist and be reconciled into one view.
The lambda architecture
Lambda runs two paths in parallel. A batch layer reprocesses the full dataset periodically, producing accurate, comprehensive views. A speed layer processes only recent events with low latency, producing approximate views for the interval the batch has not yet covered. A serving layer merges the two so a query gets accurate history plus fresh recent data. The cost is maintaining two code paths that must compute the same logic consistently.
The duplication problem
- Business logic is implemented twice, in batch and in stream code
- The two implementations drift, producing subtly different results
- Debugging requires reasoning about both paths and their merge
- Reprocessing means fixing two code bases, not one
The kappa architecture
Kappa eliminates the batch layer. Everything is a stream, and the event log is retained long enough that reprocessing means replaying the log through the same streaming code with a new version. There is one code path. To rebuild history after a logic change, you spin up a new stream job, replay from the beginning, and switch over when it catches up. Kappa depends on a durable, replayable log as its foundation.
Choosing between them
Kappa is simpler when the log can be retained and replayed and when streaming can express all the needed logic. Lambda remains relevant when some computations are genuinely batch-shaped, when historical reprocessing over enormous data is cheaper in a batch engine, or when exact recomputation must be independent of the streaming path. Many practical systems land in between. See stream processing, event streaming, and real-time analytics.