Preference Data Collection and Curation

Human preference data is collected with the same lineage rigor as machine data — who judged, on what, when — so RLHF rests on an auditable, versioned foundation.

Preferences are data, treated as data

RLHF is only as trustworthy as the preferences it learns from. Kronos collects operator and expert preferences through structured interfaces that record not just the ranking but its full context: the machine situation shown, the two responses or trajectories compared, the judge's role and expertise, timestamp, and any rationale. This is versioned and lineage-tracked like any dataset.

Judgments come from qualified operators and physicists, and are weighted by domain: a reactor physicist's ranking of a plug-stability explanation carries more weight than a generalist's, and vice versa for procedural UX. Inter-rater agreement is measured; low-agreement items are flagged for review rather than fed blindly into the reward model, because noisy preferences produce a noisy objective.

What each preference record holds

• Context snapshot: machine (breeder / burner), state, prompt
• The compared items and the chosen ranking
• Judge identity, role, and expertise weighting
• Rationale text and any correction supplied
• Agreement with other judges on the same item

pref = {
  'id':'pref:sha256:...','machine':'burner',
  'context':'state:sha256:...',
  'items':['resp_a:...','resp_b:...'],'chosen':'resp_a',
  'judge':{'id':'op-114','role':'reactor-physicist','w':1.4},
  'rationale':'cites plug stress gate correctly; hedges He-3 supply',
  'agreement':0.82
}   # append-only; feeds reward-model training set with lineage

Because preferences can encode bias, the dataset is audited for systematic skew before it trains a reward model. Golden items with known-correct answers are seeded in to catch judge drift over time. This collection discipline is the human-feedback analogue of the label provenance applied to machine data.