Copilot Evaluation Harness: Overview

A layered harness that grades copilot answers, reasoning traces, and proposals against grounded, machine-specific truth before any change ships.

Why copilots need their own harness

A copilot cannot be validated the way a numerical solver is. Its outputs are language, its reasoning is multi-step, and its failures are subtle — a fluent but ungrounded claim, a dropped safety caveat, a right answer reached by a wrong path. The evaluation harness is the machinery that catches these before a prompt, model, or retrieval change reaches operators.

What the harness grades

• Answer correctness against grounded, machine-specific reference cases
• Reasoning-trace soundness: were the tool calls appropriate and in-bounds
• Grounding: does every claim trace to a real source
• Safety: are envelope, refusal, and no-forbidden-claim rules honored
• Calibration: does stated uncertainty match observed accuracy

The harness is layered. Golden-shot cases test end-to-end behavior on curated scenarios with known-good answers. Grounding checks verify citation coverage and entailment. A regression suite runs on every change. Adversarial and red-team cases probe for failure. Each layer has its own page; this one describes how they fit together.

evaluate(candidate_copilot):
  s1 = golden_shots(candidate)        # end-to-end correctness
  s2 = grounding_checks(candidate)    # citation coverage + entailment
  s3 = safety_suite(candidate)        # envelope / refusal / claims
  s4 = calibration(candidate)         # UQ vs realized error
  s5 = red_team(candidate)            # adversarial
  ship only if all >= thresholds AND no safety regression

Grounded truth, not opinion

Reference answers are grounded in the twin, the frozen canon, and validated physics — not in a reviewer's taste. A golden shot's expected answer for the breeder is checked against twin simulation and the canonical design point (Q_sci 3.076, 85.0 MW, delta -0.30); for the burner against the tandem-mirror model (26.49 T plug, 5.44% neutron fraction). Where an LLM-as-judge is used for language quality, it is calibrated against human-labeled cases and never used to judge physical correctness.

No copilot change ships on a green aggregate score alone: a safety regression on any single case blocks the release regardless of the average. The subsequent pages cover golden shots, grounding evaluation, the regression suite, and red-teaming.