Transactional Outbox

State changes and their emitted events are committed in one local transaction, so a crash never leaves the two inconsistent.

L4 · Orchestration — its place in the stack (left, click any layer) and its internal components (right). Telemetry rises; control descends.

The dual-write problem

A consumer that updates its state and then publishes an event faces a gap: crash between the two and either the state changed without an event, or the event fired without the state change. On a control plane this could mean a procedure believes a breeder step advanced while no downstream consumer heard, or vice versa. The transactional outbox closes the gap.

One local transaction

python

with db.transaction():
    apply_state_change(cmd)                 # e.g. mark step advanced
    outbox.insert(event_for(cmd))           # SAME transaction, local
# a separate relay reads the outbox and publishes to the log at-least-once
# publish failure -> retry; downstream dedups on event_id (idempotent)

Why this beats a distributed transaction

A distributed transaction across the state store and the log would add latency and a coordinator that can itself fail. The outbox needs only a local transaction plus an idempotent relay, giving effectively-once semantics without a two-phase commit on the hot path. The relay publishes at-least-once and consumers dedup by event_id (see idempotency).

Read-process-write atomicity

Consuming an event, updating state, and enqueuing outputs commit together, so offset and state never diverge.
A crash before commit reprocesses the input; dedup absorbs the duplicate output.
This is the mechanism behind the effectively-once guarantee.

On the machines

The breeder campaign engine uses the outbox so a step advance and the command event that follows it are inseparable; an orchestrator restart mid-step resumes cleanly. The burner supervisor uses it so a state-change decision and the resulting setpoint proposal are never orphaned. The pattern underpins saga reliability.