AI Plasma Control

Reinforcement Learning for Plasma Control

Reinforcement learning trains control policies against a simulator, offering a route to coordinate many actuators toward hard-to-specify goals.

What reinforcement learning offers

Reinforcement learning (RL) trains a control policy by trial and error against a model of the system, rewarding actions that move toward a goal. For plasma control it is attractive because it can learn to coordinate many coupled actuators toward objectives that are hard to write as explicit control laws, such as achieving and holding a complex plasma shape.

Training against a simulator

Kronos motion — control room

An RL agent needs enormous numbers of trials, far more than a real machine can provide. So it is trained against a fast simulator, the same kind of reduced model used for flight simulation. The learned policy is then transferred to the real machine. This makes a fast, accurate simulator a prerequisite for RL control.

The transfer gap

A policy trained in simulation can fail on hardware if the simulator differs from reality, the so-called reality gap. Techniques to bridge it include randomizing the simulator's parameters during training so the policy is robust to model error, and fine-tuning on the real machine. Verifying a learned policy is harder than verifying a designed one, which is a live concern for safety-critical use.

Where it fits

RL is most compelling for coordination problems that are cumbersome for hand-designed controllers, and less so for the fastest safety-critical loops, where analyzable, provable behavior matters most. In practice RL policies are used within a supervisory framework that retains conventional protection, rather than replacing the whole stack.

In the Kronos program

Kronos treats reinforcement learning as a research avenue for coordinated control of the Hyperion breeder, trained against its reduced models and constrained by the same protection layer that governs all controllers. Because the machine is in simulation ahead of operation, any learned policy is validated in the flight simulator, and the fastest safety-critical loops remain conventional and analyzable.