Hindsight Experience Replay

HER turns failures into learning by relabeling trajectories with the goals that were actually achieved.

Learning from failure

In goal-conditioned tasks with sparse rewards, an agent that never reaches the goal gets almost no reward and cannot learn. Hindsight Experience Replay (HER) extracts signal from these failures: even a trajectory that missed the intended goal successfully reached some other state, and that outcome is a valid demonstration for a different goal.

Relabeling goals

HER stores each transition twice. Once with the original goal, and again with a substitute goal chosen from states actually visited later in the same episode. Under the substitute goal the agent did succeed, so it receives the success reward and learns which actions lead to that outcome. The policy and value function are conditioned on the goal, so this knowledge generalizes across goals.

# For a stored transition (s, a, s2, g):
# also add relabeled copies with g' = future achieved state
for g_prime in sample_future_states(episode, k):
    r_prime = reward(s2, g_prime)   # often 0 if goal reached, -1 otherwise
    buffer.add(s, a, s2, g_prime, r_prime)

Strategies for substitutes

• future: goals from states later in the same episode (most effective)
• final: the last state of the episode
• episode / random: states sampled more broadly

Why it works

HER converts a sparse-reward problem into one with dense, achievable targets without any reward engineering. It is an add-on to off-policy, goal-conditioned algorithms (such as DDPG or SAC with a goal input) and enabled robotic manipulation from binary success signals that were previously intractable. Its main requirement is that the reward be computable for arbitrary goals, which holds whenever the goal is defined as reaching a state.