Long-Running Procedure Checkpointing

Procedures that run for minutes to hours checkpoint their state so a restart resumes exactly, without re-energizing from scratch.

L4 · Orchestration — its place in the stack (left, click any layer) and its internal components (right). Telemetry rises; control descends.

Restarts must not restart the machine

A breeder shot lasts seconds, but a full campaign or a burner steady-state run lasts far longer, and the orchestrator process can restart mid-procedure. Checkpointing persists procedure state frequently so a restart resumes at the exact step rather than re-running an energizing sequence from the beginning, which could be unsafe on a partially-energized machine.

Checkpoint contents

json

{
  "procedure": "burner_steady_state",
  "version": "1.8.0",
  "current_step": "STEADY_STATE",
  "committed_steps": ["throat_field","plug_field","fuelling","ignition_approach"],
  "held_tokens": ["burner.plug.field","burner.throat.field"],
  "saga_compensations_pending": ["safe_bleed"],
  "last_event_offset": 918273
}

Checkpoint = last committed event

Because the procedure is a deterministic function of the event log, a checkpoint is effectively a durable offset plus derived state. On restart, orchestration reloads the checkpoint and, if needed, replays events after the checkpoint offset to reach the precise current state, reusing the replay machinery. Idempotent handlers mean any re-applied events after the checkpoint are harmless.

Token and saga continuity

Held actuator tokens are recorded so a resumed procedure re-asserts single-writer ownership.
Pending compensations are checkpointed so a crash during rollback resumes the rollback.
If a restart cannot safely re-establish state, the default is a controlled safe-state, not a blind resume.

Frequency vs cost

Checkpoint cadence is tuned so the maximum re-play distance after a restart stays small, which matters most for the burner's plug supervision where the machine must not lose the plug during an orchestrator restart. All of this is validated in simulation ahead of FOAK, with fault injection that kills the orchestrator mid-procedure.