Offline RL: Conservative Q-Learning

CQL learns from a fixed dataset by pushing down Q-values on unseen actions, preventing overestimation of out-of-distribution behavior.

The offline problem

Offline (batch) RL learns a policy from a fixed dataset of logged transitions with no further environment interaction. The central failure mode is distributional shift: the learned policy queries Q-values for actions the dataset rarely or never contains, and function approximation confidently overestimates them, producing a policy that looks great in training and fails on deployment.

The conservative fix

Kronos motion — learning physics

Conservative Q-Learning (CQL) adds a regularizer that lowers Q-values for actions the current policy would take but the data does not support, while keeping Q-values on dataset actions high. This yields a lower bound on the true value of the learned policy, so the agent cannot be fooled by optimistic errors on out-of-distribution actions.

python

# CQL objective (schematic)
# standard Bellman error + conservative penalty
cql_penalty = logsumexp(Q(s, all_actions)) - Q(s, dataset_action)
loss = bellman_error + alpha * cql_penalty.mean()

What the penalty does

The logsumexp term pushes down the peaks of the Q-function over all actions; subtracting the value on the observed action pulls that one back up. The net effect: Q stays high where there is data and is suppressed elsewhere. The learned policy therefore stays near the data support without an explicit behavior-cloning constraint.

When to use it

Data is fixed and interaction is expensive, unsafe, or impossible
The logging policy is unknown or a mixture of behaviors
You want a value-based method rather than a policy-constraint method

CQL is a strong default for value-based offline RL. Its main knob is the conservatism weight: too small and overestimation returns, too large and the policy is pinned to the data and cannot improve. Related methods instead constrain the policy directly or, like IQL, avoid querying unseen actions altogether.