RL for Continuous Control
Continuous action spaces demand policy-gradient and actor-critic methods rather than discrete value maximization.
The continuous-action challenge
Value-based methods like DQN choose actions by maximizing Q over a finite set. With continuous actions (joint torques, valve openings) that max is itself an optimization at every step, and enumerating actions is impossible. Continuous control therefore relies on policies that output actions directly and on actor-critic architectures.
Deterministic and stochastic policies
Two lineages dominate. Deterministic policy gradient methods (DDPG, TD3) output a single action and learn a critic whose gradient with respect to the action improves the actor, well suited to off-policy learning from a replay buffer. Stochastic methods (PPO, SAC) output an action distribution; PPO is on-policy and robust, while SAC is off-policy and adds an entropy bonus for exploration.
- DDPG: deterministic actor + Q-critic, off-policy, sample efficient but brittle
- TD3: DDPG plus twin critics and delayed updates to curb overestimation
- SAC: stochastic actor, entropy-regularized, strong and stable default
- PPO: on-policy, clipped updates, simple and widely used
Squashing and bounds
Real actuators have limits, so continuous policies squash outputs into bounds, commonly with a tanh applied to a Gaussian sample. This requires a correction to the log-probability when computing entropy or likelihood, a detail that matters for SAC's numerics.
# Gaussian policy with tanh squashing (bounded actions)
mu, log_std = actor(state)
u = mu + exp(log_std) * randn()
action = tanh(u) # in (-1, 1)
# log-prob correction: -sum log(1 - tanh(u)**2 + 1e-6)Where it is applied
Continuous-control RL drives robotic manipulation, locomotion, and process control. Physical plants such as fusion devices present continuous, multi-input control problems, and published work has applied deep RL to tokamak magnetic control in simulation and on experimental hardware. Kronos machines, including the Hyperion breeder, are at design and simulation stage; any learned control would be developed and validated in simulation first.