Offline-to-Online RL

Pretrain a policy on logged data, then fine-tune it with limited online interaction, combining safety and adaptability.

Best of both regimes

Pure offline RL is safe but capped by the dataset; pure online RL adapts freely but is sample-hungry and risky early on. Offline-to-online RL bridges them: learn a competent initial policy from logged data, then refine it with a modest budget of real interaction. The offline phase provides a strong, safe starting point; the online phase overcomes the dataset's limits.

The distribution-shift jolt

Naively switching from offline to online training often causes a sudden performance drop. Offline methods are trained to be conservative, staying close to the data; when fresh online data arrives from a different distribution, the value estimates can destabilize and the policy can unlearn what it knew before it recovers. Managing this transition is the central technical problem.

• Conservatism decay: relax the offline pessimism gradually as online data accumulates
• Balanced replay: mix offline and online transitions so learning is not swamped by either
• Value calibration: correct over-conservative value estimates before aggressive online updates

Exploration after pretraining

A policy pretrained offline already behaves reasonably, so online exploration should be targeted: probe where the offline data was thin and the value uncertain, rather than explore from scratch. Ensembles and uncertainty estimates carried over from the offline phase can direct this fine-tuning efficiently.

# Offline-to-online skeleton
policy = offline_train(logged_dataset)      # conservative start
buffer = replay(logged_dataset)             # seed with offline data
for step in online_budget:
    t = policy.act(env.state())             # already competent
    buffer.add(t)
    policy.update(buffer.sample(mix_offline_online))

Where it fits

Offline-to-online RL suits domains with abundant logged data and limited, costly, or risky live interaction, from robotics to control of expensive physical systems. It reflects a practical philosophy: extract everything possible from existing data first, then spend scarce real interaction only to close the remaining gap.