DAgger: Dataset Aggregation

DAgger fixes behavior cloning's drift by querying the expert on the states the learner itself visits.

The distribution mismatch, solved iteratively

Behavior cloning trains on the expert's state distribution but is tested on its own, and the mismatch causes compounding errors. DAgger (Dataset Aggregation) closes this gap by repeatedly collecting new data under the learner's own distribution and labeling it with the expert's correct actions.

The loop

• Train an initial policy by behavior cloning on the expert data
• Run the current policy in the environment to visit new states
• Ask the expert what it would do in each visited state, and record those labels
• Aggregate the new labeled states into the dataset and retrain
• Repeat

D = expert_demos
pi = train(D)
for i in range(N):
    # roll out current policy (optionally mix in expert)
    states = rollout(pi)
    labels = [expert(s) for s in states]   # expert labels learner's states
    D = D + list(zip(states, labels))
    pi = train(D)

Why it works

By training on states the learner actually reaches, DAgger removes the covariate shift that dooms plain cloning. It comes with a guarantee: under a no-regret online-learning view, the number of mistakes grows linearly in the horizon rather than quadratically. In early iterations one often mixes the expert into the rollout policy (a decaying probability) so the learner is not stranded in hopeless states before it can act competently.

Costs and variants

DAgger's price is an interactive expert: someone or something must label arbitrary learner-visited states on demand, which is expensive and sometimes infeasible (a human cannot always say the ideal action mid-trajectory). Variants reduce queries by asking only when the policy is uncertain (SafeDAgger) or by using an approximate expert. Where an interactive expert exists, DAgger is a reliable upgrade over one-shot cloning.