Successor Features and Representations

Successor representations factor value into expected future state occupancy times reward, enabling fast transfer.

Separating dynamics from reward

The successor representation (SR) decomposes value into two parts: where the agent expects to go, and how much reward each place is worth. Formally the SR M(s, s') is the expected discounted number of future visits to state s' starting from s. Then V(s) = sum over s' of M(s, s') times R(s'). If the reward changes but dynamics do not, only R must be relearned; M is reused.

Successor features

Kronos motion — safety factor

In large or continuous spaces the tabular SR is replaced by successor features. Suppose reward is linear in features: R(s) = phi(s) dot w. Define psi(s), the expected discounted sum of future features. Then value is simply psi(s) dot w. psi obeys a Bellman equation and is learned like a value function, while w captures the task. Swapping tasks means swapping w.

python

# Successor features: value = psi . w
# psi(s) = E[ sum_t gamma^t phi(s_t) ]  (Bellman-learnable)
# reward:  R(s) = phi(s) . w
V = np.dot(psi(s), w)

Transfer via GPI

Successor features shine at transfer. Given a library of policies, each with its own psi, and a new task's reward vector w, generalized policy improvement evaluates all stored policies on the new task instantly (each value is a dot product) and acts greedily with respect to the best. This produces a strong policy for a novel reward with no new learning, then refines from there.

Connections

Bridges model-free and model-based RL: SR is a partial, reward-agnostic model of dynamics
Explains some neuroscience findings on predictive maps in the hippocampus
Underlies fast multi-task and option-based transfer

Successor features are a principled route to transfer: learn the dynamics-dependent part once, and adapt only the small reward-dependent part per task.

Successor Features and Representations

Separating dynamics from reward