Model-Based RL and Dyna

Dyna blends real experience with simulated experience from a learned model to accelerate value learning.

Model-free vs model-based

Model-free methods learn a value function or policy directly from experience and discard the transitions. Model-based methods also learn a model of the transition and reward functions, then use that model to plan or to generate extra training data. Model-based RL is typically far more sample efficient, at the cost of model bias.

The Dyna architecture

Dyna-Q, introduced by Sutton, is the canonical bridge. Every real step does two things: it updates the value function directly (a model-free Q-learning step) and it updates a learned model of the world. Then, between real steps, the agent runs several planning updates using transitions sampled from the model.

# Dyna-Q core loop
for step in episodes:
    s, a = current_state, policy(current_state)
    s2, r = env.step(a)
    Q[s,a] += alpha * (r + gamma*max(Q[s2]) - Q[s,a])   # direct RL
    model[s,a] = (s2, r)                                  # learn model
    for _ in range(n_planning):                           # planning
        sp, ap = random_seen_pair(model)
        s2p, rp = model[sp, ap]
        Q[sp,ap] += alpha*(rp + gamma*max(Q[s2p]) - Q[sp,ap])

Why planning helps

Each real transition is expensive to obtain but cheap to replay through the model. The planning updates propagate reward information backward through the state space much faster than real interaction alone. Prioritized sweeping improves this further by scheduling planning updates where the value change is largest.

Limits and remedies

Dyna works cleanly when the model is accurate. In stochastic or non-stationary worlds a wrong model injects bias, so real experience must keep correcting it. Modern model-based methods extend the idea with neural dynamics models, ensembles for uncertainty, and short model rollouts branched off real states to bound error accumulation.