Champion-Challenger Evaluation

A challenger model is continuously compared against the reigning champion on identical live data, so promotion is decided by evidence, not by offline hope.

Continuous, evidence-based competition

The model currently holding authority is the champion; any candidate to replace it is a challenger. Kronos runs challengers permanently in shadow against the champion, on identical live breeder or burner data, accumulating a head-to-head record. Promotion is then a decision backed by weeks of matched, real-conditions evidence rather than by an offline test-set score alone.

The comparison is matched-pairs: on every cycle both models see the same state, so their outputs are directly comparable and confounders are removed. The evaluation tracks not just average performance but performance in the situations that matter most — disturbances, regime edges, near-fault conditions — where an average can hide a dangerous weakness.

Comparison dimensions

• Matched-pairs accuracy and calibration on identical live data
• Performance in tail situations, not just nominal operation
• Would-be envelope violations and rate-limit approaches
• Uncertainty honesty: does the challenger know when it is wrong
• Inference latency and jitter on the real edge budget

def champion_challenger(state, champ, chall, ledger):
    u_c, u_x = champ.act(state), chall.act(state)  # x is shadow only
    ledger.update(
      champ_err = later_error(u_c, state),
      chall_err = later_error(u_x, state),
      chall_violations = check_envelope(u_x, state),
      tail = is_tail_situation(state))
    # promote only if challenger wins overall AND in tail cases

A challenger is promoted to canary only if it beats the champion overall and does not lose in the tail. This guards against the common trap of a model that is better on average but worse exactly when it matters. Champion-challenger is the engine behind offline A/B evaluation and feeds the promotion decision.