Inverse Reinforcement Learning

Inverse RL recovers the reward function that best explains observed expert behavior, then optimizes it.

Inferring the objective

Standard RL is given a reward and finds a policy. Inverse reinforcement learning (IRL) reverses this: given demonstrations from an expert, it infers the reward function the expert appears to be optimizing. The recovered reward is often more transferable than the policy itself, because it can be re-optimized under new dynamics.

The ambiguity problem

IRL is fundamentally ill-posed: many reward functions, including trivial constant ones, make any policy optimal. Practical methods resolve this ambiguity with a principle. Maximum-margin IRL prefers rewards under which the expert beats alternatives by the largest margin. Maximum-entropy IRL assumes the expert is noisily optimal, making trajectory probability proportional to exp(reward), which yields a unique, well-defined distribution.

# MaxEnt IRL: P(trajectory) proportional to exp(sum of rewards)
# Learn reward weights so expert feature counts match
# the feature counts expected under the induced policy
grad = expert_feature_expectations - model_feature_expectations

Feature and reward matching

Many IRL algorithms reduce to matching feature expectations: adjust the reward so that the discounted feature counts of the learner's induced policy equal those of the expert. If features are matched, the learner's return equals the expert's under the true (linear) reward, regardless of what that reward exactly is.

Why recover the reward at all

• Rewards transfer across changed dynamics where a cloned policy would break
• A learned reward can reveal the intent behind behavior, aiding interpretability
• It supplies the objective for downstream RL when hand-specifying reward is hard

IRL underlies modern imitation methods and reward modeling. Its deep-learning descendants, such as adversarial approaches, learn a reward and policy jointly and scale to high-dimensional control.