RLHF for Operator Copilots

Operator copilots are tuned with reinforcement learning from human feedback so their suggestions match how expert Kronos operators actually reason — never to grant them control.

Aligning advice, not authority

Kronos copilots are language-and-telemetry assistants that help operators interpret machine state, draft procedures, and reason about anomalies. They advise; they never actuate. RLHF is used to make their advice trustworthy: preference data from expert operators shapes a reward model, and the copilot policy is optimized against it so its outputs align with expert judgment about what is correct, safe, and appropriately hedged.

The feedback is collected in the loop: operators rank or correct copilot responses to real breeder and burner situations, flag hallucinated numbers, and mark unsafe suggestions. Crucially, factual grounding about the machines comes from the retrieved canonical record, not from RLHF; RLHF shapes reasoning style, calibration of confidence, and refusal behavior, not the physics constants.

What the reward model rewards

• Correct, cited grounding in the canonical machine record
• Explicit uncertainty when the data is thin
• Refusal to invent numbers or overstate capability
• Respect for the honest gates when discussing the burner
• Actionable, procedurally-correct guidance for operators

# RLHF pipeline for the copilot (offline, on L0)
prefs   = collect_operator_preferences(copilot_logs)
reward  = train_reward_model(prefs)          # r_phi(prompt, response)
policy  = ppo_optimize(base=copilot, reward=reward,
                       kl_penalty=beta)       # stay near base = safety
# hard constraint: numeric claims must be retrieval-grounded, not RL'd

A KL penalty keeps the tuned policy close to the base model so RLHF cannot drift the copilot into confident nonsense. Every copilot response is still subject to the same audit and provenance rules as any output, and none of this grants the copilot a path to a machine. See reward modeling for how the reward model itself is validated, and RLHF for control policies for the far more constrained actuation case.