TD3: Twin Delayed DDPG

TD3 fixes DDPG's overestimation with twin critics, delayed policy updates, and smoothed target actions.

Three targeted fixes

Twin Delayed Deep Deterministic policy gradient (TD3) diagnoses why DDPG is unstable and applies three specific remedies. Each addresses a distinct source of error, and together they make deterministic continuous control reliable.

Clipped double-Q

Kronos motion — twin metal math

DDPG's single critic systematically overestimates values because the actor exploits its errors. TD3 learns two critics and uses the minimum of the two in the Bellman target. Taking the minimum biases estimates downward, counteracting overestimation. This is the single most important change.

Delayed policy updates

A rapidly changing critic makes the actor chase a moving target. TD3 updates the actor and the target networks less frequently than the critics, typically once every two critic updates, so the actor optimizes against a more settled value estimate. This reduces variance in the policy update.

Target policy smoothing

To prevent the actor from exploiting sharp, spurious peaks in the critic, TD3 adds clipped noise to the target action when computing the Bellman target. This regularizes the value estimate over a small neighborhood, encoding the prior that similar actions should have similar value.

python

# TD3 target with smoothing and clipped double-Q
a2 = mu_targ(s2) + clip(noise, -c, c)
a2 = clip(a2, act_low, act_high)
y  = r + gamma * min(Q1_targ(s2,a2), Q2_targ(s2,a2))
# critics -> y ; actor updated every d steps, ascending Q1(s, mu(s))

Result

TD3 substantially outperforms DDPG in both stability and final performance on continuous-control benchmarks, with little extra complexity. It shares the off-policy, replay-based structure of DDPG, so it retains high sample efficiency. TD3 and SAC are the two standard off-policy choices for continuous control; TD3 is deterministic, SAC is stochastic and entropy-regularized.