Transformer Models for Fusion Time Series
Attention-based transformers model long-range dependencies in multi-channel plasma signals for prediction and representation learning.
Attention over sequences
A transformer processes a sequence by letting each element attend to every other, weighting them by learned relevance. This captures long-range dependencies without the step-by-step recurrence of RNNs and parallelizes well during training, which suits the long multi-channel time series that fusion diagnostics produce.
Why it suits plasma data
- Long context: relate a precursor to an event many steps later
- Multi-channel fusion: attend across diagnostics simultaneously
- Parallel training: efficient on large archival datasets
- Transferable representations for pretraining and fine-tuning
Attention lets the model discover which past moments and which channels matter for a prediction, rather than compressing everything into a single recurrent state. For disruption prediction this can surface subtle multi-signal patterns that precede the event.
Costs and adaptations
Standard attention scales quadratically with sequence length, which is costly for long high-rate signals. Efficient attention variants and patching, which group samples into tokens, reduce this cost. Positional encodings inform the model about timing, which is essential for physical signals.
Pretraining
Transformers underpin the foundation-model direction in plasma physics: pretrain on large unlabeled signal archives with masked-prediction objectives, then fine-tune on specific tasks with limited labels. Whether this transfers as well as in language remains an open research question given smaller, more heterogeneous data.
For concepts like the Hyperion breeder, transformer pipelines are developed on simulated and archival data ahead of construction. They process modeled signals because the machine is not built, and any predictive claim is a computational estimate to be validated on operating hardware.