RL from Human Feedback

RLHF learns a reward model from human preference judgments, then optimizes a policy against it to align behavior with human intent.

Rewards from preferences

Reinforcement learning from human feedback (RLHF) is a method for training agents when the goal is hard to write as a reward but people can judge which of two behaviors is better. Instead of a hand-coded reward, RLHF learns a reward model from human preference comparisons and then optimizes a policy against it.

The pipeline

Kronos motion — pid vs model

Collect comparisons: humans see pairs of behaviors or outputs and pick the preferred one.
Train a reward model: fit a model that scores behavior consistently with those preferences.
Optimize the policy: use an RL algorithm, commonly PPO, to maximize the learned reward.
Constrain drift: penalize moving too far from the starting policy to preserve competence and avoid degenerate solutions.

Why preferences

Absolute reward labels are noisy and hard for people to give consistently, but relative judgments — this is better than that — are easier and more reliable. The reward model turns many such comparisons into a smooth scoring function that generalizes to behaviors no human directly rated. This connects to inverse RL, which likewise infers reward from human data.

Failure modes

RLHF inherits the risks of any learned reward. The policy can overoptimize the reward model, exploiting its errors to score highly in ways humans would not actually endorse — a form of reward hacking. Guarding against this requires the KL penalty to the original policy, careful data collection, and ongoing evaluation against the true intent rather than the proxy.

Significance

RLHF became a standard technique for aligning the behavior of large models with human preferences, precisely because it targets goals that resist explicit specification. It is a clear example of RL's reach beyond control into shaping the behavior of complex systems from human judgment.