Reinforcement-Learning Basics
The core concepts of learning to act, framed for readers who will meet RL in plasma control.
The setting
Reinforcement learning studies an agent that observes a state, takes an action, receives a reward, and lands in a new state. The goal is a policy, a mapping from states to actions, that maximizes cumulative reward over time. This frames control, where actions steer a dynamic system toward objectives.
Key quantities
- Policy: what action to take in each state
- Value function: expected future reward from a state
- Reward: the scalar signal defining the objective
- Discount factor: how much future reward counts now
Families of methods
Value-based methods learn value functions and act greedily with respect to them. Policy-gradient methods adjust the policy directly by following the gradient of expected reward. Actor-critic methods combine both, using a learned value estimate to reduce the variance of policy updates. Continuous-action control, as in coil voltages, typically uses policy-gradient or actor-critic methods.
Exploration versus exploitation
An agent must try new actions to discover better ones (exploration) while using what it knows to earn reward (exploitation). Balancing these is central, and in physical systems exploration must be bounded so it never drives the system into unsafe states.
Why RL is delicate
RL can be sample-hungry, sensitive to reward design, and prone to exploiting quirks of the simulator rather than solving the real task. A poorly specified reward yields a policy that maximizes the wrong thing. For safety-critical control, these properties demand careful reward design, extensive simulation testing, and hard constraints on actions.