Reinforcement Learning for Shape Control
Training control policies in simulation to steer plasma shape and position, and the honest limits of the approach.
Learning a policy
Reinforcement learning (RL) trains a controller - a policy mapping measured state to actuator commands - by rewarding it for keeping the plasma on target in a simulator. Rather than hand-designing gains from a linear model, the agent discovers a control law by trial and error against a plasma model, then that fixed policy is deployed to run the loop.
Why it drew interest
Demonstrations on research tokamaks showed a single learned policy holding a range of plasma shapes and even novel configurations, coordinating all coils at once. RL can, in principle, learn the coupled multivariable control that is tedious to design by hand, and can optimize directly for the true objective rather than a linearized proxy.
The simulator dependency
RL for tokamak control is trained almost entirely in simulation, because real discharges are too few and too costly to explore by trial and error. This makes the approach only as good as the simulator: a policy that exploits an inaccuracy in the model will fail on the real plasma. Bridging this sim-to-real gap is the central difficulty.
Honest limits
- A learned policy is hard to certify - its behavior outside training states is not guaranteed
- Extrapolation to unseen regimes is unreliable, and reactor regimes are unseen
- Verification and safety cases are harder than for a transparent linear law
- It must still sit above the deterministic safety layers, never replace them
Where it fits
RL is a promising tool for the shape and scenario layers, used as decision support or as a control law wrapped inside conventional safety limits and monitored by state estimation. At Kronos any such use is design and simulation study, not deployed on hardware; the fast safety-critical reflexes remain deterministic and verifiable regardless of what learns above them.