Deterministic Policy Gradients and DDPG

DDPG learns a deterministic continuous-control policy off-policy, using a critic's action gradient to improve the actor.

A deterministic actor

The deterministic policy gradient theorem shows that for a deterministic policy mu(s), the gradient of expected return is the expected gradient of the critic Q with respect to the action, chained through the actor: grad J = E[ grad_a Q(s,a) at a=mu(s) times grad_theta mu(s) ]. This avoids integrating over an action distribution and enables efficient off-policy learning.

The DDPG algorithm

Deep Deterministic Policy Gradient (DDPG) is the deep-learning realization. It maintains four networks: an actor mu and critic Q, each with a slowly updated target copy. The critic is trained by the usual Bellman error using the target networks; the actor is trained by ascending the critic's action gradient. Experience is stored in a replay buffer and sampled off-policy, as in DQN.

# DDPG updates per batch
y = r + gamma * Q_targ(s2, mu_targ(s2))          # critic target
critic_loss = mse(Q(s, a), y)
actor_loss  = -Q(s, mu(s)).mean()                # ascend critic
# soft-update targets: theta_targ = tau*theta + (1-tau)*theta_targ

Exploration

Because the policy is deterministic, exploration comes from adding noise to actions during data collection, originally temporally correlated Ornstein-Uhlenbeck noise, though uncorrelated Gaussian noise usually works as well. Target networks and the replay buffer are essential for stability, mirroring the tricks that made DQN work.

Limitations

DDPG is sample efficient but notoriously brittle: it is sensitive to hyperparameters and prone to overestimation bias, where the critic's errors are amplified by the max-like actor update, sometimes causing collapse. TD3 was designed specifically to fix these issues, and SAC offers a more stable stochastic alternative. DDPG remains the conceptual foundation for both.