Message Queues
A message queue decouples producers from consumers by buffering messages, absorbing load spikes and letting components fail and recover independently.
Decoupling producers and consumers
When one service calls another directly, the caller's speed is bound to the callee's, and a downstream outage propagates upstream. A message queue breaks this coupling: producers write messages to the queue and move on; consumers read at their own pace. The queue is a buffer that absorbs bursts and keeps each side ignorant of the other's availability.
Delivery semantics
The hardest questions in queuing concern how many times a message is delivered. Three guarantees exist: at-most-once (may lose messages, never duplicates), at-least-once (never loses, may duplicate), and exactly-once (neither, but expensive and often only within one system's boundary). Most practical systems provide at-least-once and require consumers to be idempotent so duplicates cause no harm.
Acknowledgements
A consumer signals successful processing with an acknowledgement. Until it acks, the message is considered in-flight and will be redelivered if the consumer crashes. Acking before processing risks loss; acking after processing risks duplication on a crash between processing and ack. This choice is exactly the at-most-once versus at-least-once trade-off made concrete.
Queue versus log
- A classic queue removes a message once a consumer acks it
- A log-based system retains messages and tracks each consumer's offset
- Queues suit work distribution among competing workers
- Logs suit replay and multiple independent consumers of the same stream
Dead-letter handling
A message that repeatedly fails processing must not block the queue forever. After a bounded number of retries it is routed to a dead-letter queue for inspection, so one poison message does not stall healthy traffic. Monitoring the dead-letter queue is how operators discover systematic processing bugs. See event streaming and stream processing.