Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is RLHF? Reinforcement learning from human feedback

RLHF trains a reward model on human preference rankings, then tunes an LLM to score well on it without drifting far. The three steps, with a Python demo.

What is RLHF? Reinforcement learning from human feedback

RLHF (reinforcement learning from human feedback) is a post-training method that teaches a language model what people prefer: humans rank pairs of model answers, a reward model learns to predict those rankings, and the language model is then trained with reinforcement learning to produce answers the reward model scores highly, with a penalty for drifting too far from where it started. It’s the step that turned GPT-3 into InstructGPT.

I ran the demo below on 2026-10-31 with Python 3.13.11 and no packages.

Why do language models need RLHF?

A pretrained model predicts the next token of internet text. That isn’t the same as doing what a user asked. The InstructGPT paper (Ouyang et al., OpenAI, March 2022) opens on exactly this point: “Making language models bigger does not inherently make them better at following a user’s intent.” Large models can be “untruthful, toxic, or simply not helpful”.

Supervised fine-tuning (SFT) on written demonstrations helps, but demonstrations are expensive and only show one good answer. Comparisons are cheaper and carry more signal: it’s easier for a person to say “A is better than B” than to write the ideal answer. RLHF turns those comparisons into a training signal.

The headline result made the case. In OpenAI’s human evaluations, outputs from the 1.3B-parameter InstructGPT were preferred to outputs from the 175B GPT-3, “despite having 100x fewer parameters”. I covered how that recipe reached the public in the ChatGPT research preview; this post is about the method itself, which sits in the post-training stage of how LLMs are trained.

How does RLHF work?

The InstructGPT paper lays out three steps.

  1. Supervised fine-tuning. Labelers write demonstrations of good behavior for prompts, and the pretrained model is fine-tuned on them. The SFT dataset had about 13k training prompts.
  2. Reward model. The SFT model produces several answers per prompt. Labelers rank them, anywhere from K = 4 to K = 9 at a time, and every pair from a ranking becomes a training example. A reward model is trained to give the preferred answer a higher score, with the loss -log σ(r(x, y_w) - r(x, y_l)). That’s a Bradley-Terry model of pairwise preference. The RM dataset had 33k training prompts, and OpenAI used 6B reward models, noting that 175B reward model training “could be unstable”.
  3. Reinforcement learning. The SFT model is optimized with PPO (Schulman et al., 2017) to maximize the reward model’s score, minus “a per-token KL penalty from the SFT model” to “mitigate overoptimization of the reward model”. The PPO dataset had 31k prompts, from the API only.

The labelers were “a team of about 40 contractors” hired through Upwork and Scale AI. The idea predates LLMs: Christiano et al. (2017) trained RL agents on Atari and simulated robots from human comparisons of short clips, with feedback on less than 1% of the agent’s interactions. OpenAI applied it to summarization in 2020 before InstructGPT.

What does the KL penalty do?

It’s the leash. The objective in the paper is reward minus β log(π_RL(y|x) / π_SFT(y|x)), summed into a KL divergence over the policy’s samples. Without it, the policy would find whatever text the reward model overrates and produce only that.

The DPO paper derives that this objective has a closed-form optimal policy: π*(y|x) ∝ π_ref(y|x) * exp(r(x, y) / β). PPO is an expensive way to approximate that distribution with a neural network. For a toy problem with five possible answers, I can compute it exactly, which makes the effect of β visible.

What does RLHF look like in code?

The script fits a reward model on 4,000 simulated preference pairs and then computes the KL-regularized policy at several β values. The labelers in the simulation judge only true quality. The reward model can’t see quality directly: it sees a noisy signal of quality plus the answer’s length, and in this data better answers also tend to be longer. That’s a deliberately planted shortcut, and the question is whether optimization finds it.

# rlhf.py: the RLHF loop in miniature. Fit a reward model on pairwise preferences,
# then compute the KL-regularized policy and watch it over-optimize.
# Python 3 standard library only.
import math, random

random.seed(42)
sigmoid = lambda x: 1 / (1 + math.exp(-x))

# 1. Preference data. Labelers judge true quality only. The reward model sees
#    a noisy signal of quality plus the answer's length, and in this data longer
#    answers tend to be better, so length becomes a shortcut.
def sample_answer():
    quality = random.gauss(0, 1)
    signal = quality + random.gauss(0, 1)          # what the reward model can "see"
    length = 0.6 * quality + random.gauss(0, 0.8)
    return quality, signal, length

pairs = []
for _ in range(4000):
    a, b = sample_answer(), sample_answer()
    a_wins = random.random() < sigmoid(2 * (a[0] - b[0]))   # Bradley-Terry on quality only
    c, rj = (a, b) if a_wins else (b, a)
    pairs.append(((c[1], c[2]), (rj[1], rj[2])))             # RM gets (signal, length)

# 2. Reward model r = w_q * signal + w_len * length, trained with the
#    InstructGPT loss: -log sigmoid(r(chosen) - r(rejected)).
w = [0.0, 0.0]
for epoch in range(200):
    g = [0.0, 0.0]
    for (cq, cl), (rq, rl) in pairs:
        p = sigmoid(w[0] * (cq - rq) + w[1] * (cl - rl))
        g[0] += (1 - p) * (cq - rq); g[1] += (1 - p) * (cl - rl)
    w = [wi + 0.5 * gi / len(pairs) for wi, gi in zip(w, g)]
print(f"reward model weights: quality signal {w[0]:.2f}, length {w[1]:.2f}")

# 3. One prompt, five candidate answers: (name, true quality, length, SFT probability).
#    To be generous to the reward model, it sees each answer's quality without noise.
cands = [("concise, correct", 1.5, 0.0, 0.25), ("thorough, correct", 1.5, 1.0, 0.20),
         ("partly right",     0.5, 0.5, 0.30), ("wrong",            -1.0, 0.0, 0.20),
         ("padded, vague",    0.2, 4.0, 0.05)]
r = [w[0] * q + w[1] * l for _, q, l, _ in cands]
for (name, *_), ri in zip(cands, r):
    print(f"  RM score {ri:+.2f}  {name}")

# 4. The RLHF objective max E[r] - beta * KL(pi || pi_sft) has the closed-form
#    optimum pi(y) proportional to pi_sft(y) * exp(r(y) / beta). PPO approximates it.
print(f"\n{'beta':>5} {'E[RM score]':>12} {'E[true quality]':>16} {'KL':>6}  most likely answer")
for beta in (10, 2, 1, 0.5, 0.25, 0.1):
    un = [ref * math.exp(ri / beta) for (*_, ref), ri in zip(cands, r)]
    pi = [u / sum(un) for u in un]
    er = sum(p * ri for p, ri in zip(pi, r))
    eq = sum(p * q for p, (_, q, _, _) in zip(pi, cands))
    kl = sum(p * math.log(p / ref) for p, (*_, ref) in zip(pi, cands))
    top = max(range(len(pi)), key=pi.__getitem__)
    print(f"{beta:>5} {er:>12.2f} {eq:>16.2f} {kl:>6.2f}  {cands[top][0]} ({pi[top]:.0%})")
reward model weights: quality signal 0.53, length 0.49
  RM score +0.80  concise, correct
  RM score +1.29  thorough, correct
  RM score +0.51  partly right
  RM score -0.53  wrong
  RM score +2.06  padded, vague

 beta  E[RM score]  E[true quality]     KL  most likely answer
   10         0.65             0.68   0.00  partly right (30%)
    2         0.83             0.84   0.06  partly right (27%)
    1         1.04             0.93   0.21  thorough, correct (32%)
  0.5         1.42             0.86   0.77  padded, vague (39%)
 0.25         1.90             0.43   2.15  padded, vague (82%)
  0.1         2.06             0.20   2.98  padded, vague (100%)

The reward model learned to put almost as much weight on length as on quality, so it scores the padded, vague answer highest. Now read the β table from top to bottom. As the leash loosens, the reward model’s score climbs every row, from 0.65 to 2.06. True quality rises at first, peaks at 0.93 around β = 1, then falls to 0.20 as the policy collapses onto the padded answer.

That curve is Goodhart’s law, and it has a measured version. Gao, Schulman and Hilton (2022) used a “gold-standard” reward model as stand-in humans and showed that optimizing a proxy reward model too hard “can hinder ground truth performance”. In their setup, the relationship follows a smooth functional form whose coefficients scale with reward model size.

Where does RLHF break?

  • Reward hacking. My toy is the simple case. Real policies find longer answers, confident tone, flattery or formatting that the reward model likes. DeepSeek’s R1 paper avoids neural reward models for reasoning tasks for this reason, saying they’re “susceptible to reward hacking during large-scale reinforcement learning”. That’s the motivation for RLVR.
  • The alignment tax. InstructGPT lost some performance on public NLP benchmarks after RLHF. The fix was PPO-ptx: mixing pretraining gradients into the PPO update, which the paper says reduced the regressions “without compromising labeler preference scores”.
  • Whose preferences. About 40 contractors defined “better” for InstructGPT. The reward model inherits their instructions, judgments and blind spots.
  • Cost and instability. PPO-based RLHF keeps a policy, a frozen reference, a reward model and a value function in play at once. InstructGPT initialized the value function from the reward model. Getting this stable at scale takes real engineering, which is why simpler methods appeared.

What replaced or extended RLHF?

Three lines of work grew out of it.

  • DPO removes the reward model and the RL loop and optimizes the policy directly on preference pairs. Meta’s Llama 3 paper says they explored PPO “but found that DPO required less compute for large-scale models and performed better”. See DPO vs RLHF.
  • RLAIF and Constitutional AI. Anthropic’s Constitutional AI paper (December 2022) replaced human harmlessness labels with AI preference judgments guided by a list of principles, calling it “RL from AI Feedback”.
  • RLVR swaps the learned reward model for a program that checks the answer, which suits math and code. That’s how DeepSeek R1 trained its reasoning.

What I would do

I wouldn’t run PPO-based RLHF as an application team. The engineering cost is high and the failure modes are subtle. If I needed to shape a model’s behavior from preference data, I’d start with DPO on a strong open-weight model, and only consider an online RL method once I had a reward I trusted.

The lesson I’d keep from RLHF applies to every eval and judge I build: whatever scores the output becomes the target. If I use an LLM as a judge, I check it against human labels and watch for the judge’s shortcuts, length first. The β table above is a good reminder that a rising score and a better product can diverge.

FAQ

What is the difference between RLHF and fine-tuning?

Supervised fine-tuning trains the model to imitate given answers. RLHF trains it to produce answers that score well under a reward model learned from human preferences. In InstructGPT, SFT is the first of RLHF’s three steps.

How many people label data for RLHF?

For InstructGPT, “a team of about 40 contractors” wrote demonstrations and ranked outputs. The paper reports that the models also generalized to the preferences of held-out labelers who produced no training data.

What is a reward model in RLHF?

A model, usually initialized from a language model, that takes a prompt and an answer and outputs one score. It’s trained on human rankings so the preferred answer in each pair gets the higher score.

Why does RLHF use a KL penalty?

To stop the policy from drifting far from the SFT model into text the reward model overrates. InstructGPT adds a per-token KL penalty from the SFT model “to mitigate overoptimization of the reward model”.

Is RLHF still used?

In some form, yes, but the pieces changed. Many open recipes use DPO for preferences, and reasoning models add reinforcement learning with verifiable rewards. The idea of learning from comparisons stays.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.