Preventing Training-Serving Skew

Skew — any difference between how a feature or model behaves in training versus production — is treated as a defect and caught by contract tests before deployment.

The quiet killer of deployed models

A model can score perfectly offline and still misbehave on the machine if the production pipeline computes its inputs differently, orders them differently, or handles missing values differently than the training pipeline did. Kronos classifies every such divergence as skew and treats it as a release-blocking defect, not a tolerable quirk.

Three sources of skew

• Feature skew: a feature computed differently offline vs online
• Distribution skew: production inputs from a different regime (see covariate shift)
• Serving skew: differences in batching, dtype, quantization, or timing

The feature store removes most feature skew by construction. Serving skew is caught by a mandatory contract test: a fixed set of recorded machine states is passed through both the offline model and the compiled online artifact, and the outputs must match within tolerance. A mismatch quarantines the artifact.

def parity_check(offline_model, online_artifact, probes, tol):
    for s in probes:                       # recorded machine states
        a = offline_model.predict(s)
        b = online_artifact.infer(s)       # compiled edge form
        assert max_abs(a - b) <= tol, f'serving skew on {s.id}'
    return 'PARITY_OK'   # required gate before SHADOW

Serving skew matters most where an artifact is compiled and quantized for the L1 edge, because quantization can shift outputs. The parity check runs on the exact compiled form that will execute on the machine, so what is validated is what runs. This gate sits inside the broader validation gate set.