Deep Deterministic Policy Gradient (DDPG)

DDPG is an off-policy actor-critic method for continuous control, learning a deterministic policy alongside a Q-function critic.

Continuous-action Q-learning

Deep Deterministic Policy Gradient (DDPG) extends the ideas of DQN to continuous action spaces. Q-learning's max over actions is intractable when actions are real-valued vectors, so DDPG learns a deterministic actor that outputs the action believed to maximize the critic's Q value, sidestepping the explicit maximization.

Actor and critic

Kronos motion — actor critic

The critic Q(s, a; w) is trained with a Q-learning-style bootstrapped target, using the actor to choose the next action. The actor mu(s; theta) is trained to output actions that maximize the critic, following the gradient of Q with respect to the action, propagated back into the actor parameters. This is the deterministic policy gradient.

Borrowing from DQN

Experience replay: an off-policy buffer decorrelates data and reuses it.
Target networks: slowly updated copies of actor and critic stabilize the bootstrapped target, here using soft (Polyak) updates that blend a small fraction of the new weights each step.

Exploration

Because the actor is deterministic, exploration must be added externally by perturbing the chosen action with noise, historically an Ornstein-Uhlenbeck process and later simple Gaussian noise. The scale of this noise controls the exploration-exploitation balance.

Limitations

DDPG can be sample efficient but is notoriously sensitive to hyperparameters and prone to overestimating Q values, which can destabilize learning. These weaknesses motivated TD3, which adds twin critics and delayed updates, and SAC, which adds entropy-based exploration and stochastic policies. For simulated continuous-control tasks, these successors are usually preferred.