A/B and Interleaving Evaluation Offline

Since live A/B on a single machine is unsafe, Kronos compares models by counterfactual replay on the twin and by matched shadow evaluation, not by splitting live traffic.

No live traffic split on one machine

In software, A/B testing splits live traffic between two versions. A fusion machine is a single physical system where you cannot run two controllers on two halves of the plasma. Kronos therefore replaces live A/B with two safe substitutes: counterfactual replay on the digital twin, and matched-pairs shadow evaluation on the real data path.

Counterfactual replay takes recorded machine states and rolls each candidate forward through the calibrated twin, producing a comparable outcome for each without touching hardware. Because both candidates start from identical recorded initial conditions, the comparison is a clean paired experiment. Its validity rests entirely on twin fidelity, so replay results are always cross-checked against shadow evidence on the real machine.

The two methods

• Twin counterfactual replay: same initial states, compare rolled-out outcomes
• Matched shadow evaluation: both models see identical live inputs
• Interleaving for copilots: alternate suggestions, collect operator preference
• Statistical care: paired tests, sequential analysis, stopping rules
• Cross-validation: twin replay must agree with shadow evidence

def counterfactual(twin, states, A, B):
    dA, dB = [], []
    for s0 in states:                    # identical starts -> paired
        dA.append(twin.rollout(s0, policy=A))
        dB.append(twin.rollout(s0, policy=B))
    return paired_test(score(dA), score(dB))   # e.g. Wilcoxon

For copilots, which do not actuate, a gentler interleaving is possible: alternate which model's suggestion is shown and record operator preference, feeding preference data. For controllers, replay plus champion-challenger shadow is the standard, and both depend on a well-refined twin, tying this method back to twin refinement.