Workflow & Campaign Engine

Shot and experiment procedures run as durable, resumable state machines that survive process restarts without losing their place.

Procedures as durable state machines

A breeder shot or a burner startup is a multi-step procedure spanning seconds to hours: pump down, field ramp, fuelling, heating, diagnostics windows, ramp-down, and post-shot recovery. The campaign engine models each as an explicit, persisted state machine. Every transition is an event on the backbone, so a crashed orchestrator resumes exactly where it stopped rather than restarting a partially energized machine.

Determinism requirement

Procedure logic is deterministic: given the same event history it takes the same transitions. All nondeterminism (twin predictions, sensor reads, timers) enters only as recorded events. This is what makes the engine replayable and testable against the digital twin before any hardware exists.

# procedure code is deterministic; side effects go through the engine
@procedure
def breeder_shot(ctx):
    ctx.step("pumpdown",      until=lambda s: s.base_pressure_ok)
    ctx.step("field_ramp",    cmd=ramp_toroidal_field(target_T=8.0))   # on-axis
    ctx.gate("go_no_go", approver="session_leader")   # human-in-the-loop
    ctx.step("fuelling",      cmd=gas_program(shot=ctx.shot_id))
    ctx.step("heating",       cmd=aux_heating_program())
    ctx.step("diagnostics",   window_ms=ctx.plan.diag_window)
    ctx.compensate_on_fault(safe_rampdown)   # saga guard

What the engine guarantees

• Exactly-once step execution via idempotent commands and durable step markers.
• Automatic compensation on fault, so a half-run procedure leaves the machine in a safe state.
• Human gates as first-class steps that block until approved (see approval routing).
• Full lineage: every step, its inputs, and the twin/copilot proposals that shaped it are recorded.

Breeder vs burner cadence

The breeder (Hyperion) runs discrete pulsed shots, so its procedures are shot-scoped and repeat with parameter variation across a campaign. The burner (Aegis / MetroVolt) targets steady-state operation, so its procedures look more like long-running supervisory loops with rare state changes and continuous plug supervision. The same engine expresses both; only the step cadence differs.

Long procedures checkpoint frequently; see checkpointing and saga and compensation.