Production Model Monitoring

A model with authority is watched every cycle — inputs, outputs, uncertainty, latency, and agreement with physics — so degradation is caught before it becomes a fault.

Deployment is the start, not the end

Once a model holds authority over the breeder or burner, monitoring is continuous and non-negotiable. Every cycle, monitors observe the model's inputs, outputs, self-reported uncertainty, inference latency, and agreement with independent physics checks. Monitoring exists to catch the moment a model that passed all gates begins to fail in the field, and to arm rollback before failure becomes fault.

Monitoring is layered. Input monitors watch for covariate shift and out-of-envelope operation. Output monitors watch for physically implausible commands, sudden distribution changes, and rate-limit approaches. Behavioral monitors compare the model against a cheap independent predictor and against the physics-based twin; large, sustained disagreement is a red flag even when each individual output looks plausible.

Monitored signals

• Input distribution and envelope membership
• Output plausibility, saturation, and rate-limit proximity
• Model self-reported uncertainty and abstention rate
• Inference latency and jitter against the edge budget
• Agreement with physics twin and with the incumbent shadow

def monitor_cycle(model, state, out, twin, log):
    flags = {
     'ood':       not model.envelope.contains(state),
     'implausible': not physics_plausible(out, state),
     'unc_spike': model.uncertainty(state) > TH.unc,
     'latency':   model.last_latency > EDGE_BUDGET,
     'twin_div':  norm(out - twin.expected(state)) > TH.div}
    log.record(state.t, out, flags)
    if any(flags.values()): arm_rollback(model, flags)

Monitoring feeds three consumers: rollback, which acts immediately on a breach; the continual-learning loop, which schedules retraining on slow degradation; and the human operators, who see monitor state in the control room. A model in production is never trusted blindly; it is trusted while watched.