Reward Shaping for Plasma Control
Designing the reward signal so a learned controller pursues the physical goal and not a loophole.
The reward defines the task
In reinforcement learning the reward is the entire specification of what the agent should do. If the reward rewards the wrong thing, the agent optimizes the wrong thing, sometimes with impressive competence. For plasma control this makes reward design a physics and engineering task, not a formality.
Multi-objective rewards
Plasma control balances several goals at once: match the target boundary shape, hold plasma current, keep the position vertically stable, respect coil-current and voltage limits. These are combined into one reward, often as a weighted sum of per-objective terms, with weights that encode their relative priority.
- Tracking terms for shape, position, and current
- Penalty terms for constraint violations
- Smoothness terms to discourage actuator chatter
Reward hacking
Agents find loopholes. A reward that only measures average shape error might be met by a policy that oscillates wildly but averages out, or that parks the plasma in an easy but useless configuration. Guarding against this means testing the learned behavior, not just the reward number, and adding terms that close the loopholes.
Shaping versus sparse rewards
A sparse reward (success or failure at the end) gives little learning signal. Shaped rewards provide dense, informative feedback at every step, speeding learning, but a badly shaped reward can bias the policy toward a suboptimal strategy. The art is dense feedback that still points at the true objective.
Practice
Reward terms are tuned in simulation, inspected by plotting the resulting trajectories, and cross-checked with control engineers who know what safe, effective behavior looks like. The final policy is judged by its plasma behavior, not by its reward score alone.