Batch Retraining Pipeline

The scheduled L0 loop that re-grounds every twin surrogate against fresh pulse histories so the real-time shadow does not drift.

Why retraining is a standing process

A surrogate trained once will drift as the machine and its diagnostics change. The batch retraining pipeline is the L0 process that periodically re-fits every twin model, PINNs, GNNs, surrogates, and anomaly ensembles, against the latest accumulated pulse histories. It is scheduled, versioned, and validated, not ad hoc, because the twin's trustworthiness depends on it.

The loop

Each cycle pulls curated data from the petabyte archive, retrains or fine-tunes the affected models on the GPU clusters, validates them against held-out pulses and physics checks, and, only on passing, promotes them for compilation to the real-time twin. Failing candidates are rejected and the prior model stays in service.

• Ingest curated pulses and simulation campaigns from the archive
• Retrain or fine-tune affected surrogates on GPU clusters
• Validate against held-out data and physics residuals
• Promote, version, and stage for twin compilation

Cadence and triggers

Retraining runs on a regular cadence and also on triggers: a new simulation campaign, a diagnostic recalibration, or drift flagged by the twin against reality. Because it runs offline on L0, it can take the time to do full validation, which is exactly why this work is separated from the real-time layers that cannot pause to retrain.

The pipeline serves both machines. Breeder equilibrium and disruption models and burner ambipolar-potential and DEC models are retrained on the same infrastructure, each against its own machine's data. A single loop keeps both twins current without duplicating machinery.

Every promotion is recorded with the data version, code version, and validation report, so any deployed twin model can be traced back to the exact data and run that produced it. Retraining is thus not only about accuracy but about maintaining an auditable lineage from live model back to certified L0 ground truth.