Quantile Regression DQN

QR-DQN represents the return distribution by learned quantile locations, removing C51's fixed value grid.

Flip the axes

C51 fixes a set of return values (the support) and learns their probabilities. Quantile Regression DQN (QR-DQN) does the opposite: it fixes a set of probabilities (uniform quantile fractions) and learns the return values at those quantiles. This removes the need to guess V_min and V_max and lets the support adapt to the task.

Learning quantiles

The network outputs N values, one per quantile fraction tau_i = (i - 0.5)/N. Together they approximate the inverse cumulative distribution of the return. Because the estimates are quantiles, the natural training loss is the quantile (pinball) loss, made smooth near zero via the Huber function to give quantile Huber loss.

def quantile_huber(td_error, tau, kappa=1.0):
    huber = np.where(np.abs(td_error) <= kappa,
                     0.5*td_error**2,
                     kappa*(np.abs(td_error) - 0.5*kappa))
    return np.abs(tau - (td_error < 0).astype(float)) * huber / kappa

Wasserstein view

Minimizing the quantile loss corresponds to reducing the 1-Wasserstein distance between the predicted and target return distributions, the very metric under which the distributional Bellman operator is a contraction. QR-DQN thus aligns the practical loss with the theory, something C51's cross-entropy did not do exactly.

Consequences

• No fixed value range; the support is learned and unbounded
• A principled loss tied to the distributional Bellman contraction
• Foundation for implicit quantile networks (IQN), which sample tau continuously

QR-DQN and C51 are the two pillars of distributional value learning. C51 came first and proved the idea's power; QR-DQN refined the representation. Both feed risk-aware policies and both improve stability enough to be standard ingredients in strong deep-RL agents.