Evaluation: Regression and Continuous Testing

Every prompt, model, retrieval, or tool change runs the full evaluation suite; a safety regression on any case blocks the release.

Copilots change; the contract must not regress

The pieces that shape copilot behavior change often — a new model version, a reworded prompt, a re-embedded index, an added tool. Each change risks a silent regression. The regression suite runs the full evaluation — golden shots, grounding, safety, calibration, adversarial — on every candidate change and compares against the current production baseline before anything ships.

Gating policy

gate(candidate, baseline):
  r = evaluate(candidate)
  if r.safety_failures > 0:               return BLOCK  # hard
  if r.grounding < baseline.grounding - eps: return BLOCK
  if r.correctness < baseline.correctness - eps: return BLOCK
  if r.calibration_error > cal_max:       return BLOCK
  else:                                   return ALLOW(with review)

What triggers a run

• Model or model-version change
• Prompt-template edit (any section, including wording)
• Embedding model or re-embed of the index
• New or modified bounded tool, or a schema change
• Twin or canon update that changes reference answers

Safety is a hard gate, not a weighted term. A single safety regression — an answer that leaves the envelope, a dropped refusal, a forbidden claim, an economics leak — blocks the release regardless of how much correctness improved elsewhere. This asymmetry is deliberate: fluency gains never justify a safety loss.

Reproducibility

Every evaluation run is reproducible: the model version, prompt version, index snapshot, twin version, and random seeds are recorded, so a result can be re-derived exactly. This ties into the platform's determinism guarantees on L0 and the audit record. When the frozen canon or the twin is updated, affected reference answers are regenerated and re-reviewed rather than silently accepted.

The suite runs in continuous integration for copilot changes and on a schedule against live retrieval to catch data-drift regressions. Results feed a dashboard the copilot owners watch, and a trend down in grounding or calibration triggers investigation before it reaches operators. See the harness overview and red-teaming.