DPO vs RLHF: what's the difference, and which to use?
DPO vs RLHF: DPO trains on preference pairs with one classification loss, no reward model or PPO loop. How the two compare, with a Python demo that checks it.

DPO (Direct Preference Optimization) and RLHF solve the same problem, tuning a language model to match human preferences, but RLHF does it by training a separate reward model and then running reinforcement learning against it, while DPO skips both and trains the model directly on preference pairs with a single classification-style loss. Under the standard assumptions, both target the same optimal policy; DPO just gets there with less machinery.
I ran the demo below on 2026-10-31 with Python 3.13.11 and no packages.
What is the difference between DPO and RLHF?
Start with what they share. Both take a supervised fine-tuned (SFT) model and a dataset of preference pairs: a prompt, a chosen answer and a rejected answer. Both want to raise the probability of answers people prefer without drifting far from the SFT model.
RLHF, as in the InstructGPT paper, does it in two learned stages:
- Train a reward model with the loss
-log σ(r(x, y_w) - r(x, y_l)), so the chosen answery_wscores higher than the rejectedy_l. - Run PPO: sample answers from the current policy, score them with the reward model, subtract a KL penalty against the SFT model, and update.
DPO (Rafailov, Sharma, Mitchell, Ermon, Manning and Finn, May 2023) collapses this into one stage. The paper’s subtitle is the argument: “Your Language Model is Secretly a Reward Model”. The abstract calls RLHF “a complex and often unstable procedure” and says DPO solves the same problem “with only a simple classification loss”, “eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning”.
| RLHF (PPO) | DPO | |
|---|---|---|
| Reward model | Trained separately | None; implicit in the policy |
| Sampling during training | Yes, every PPO step | No, trains on a fixed dataset |
| Models in play | Policy, reference, reward model, value function | Policy, reference |
| Loss | RL objective with KL penalty | Logistic loss on pairs |
| Key knob | KL coefficient β | β, same role |
How does DPO work?
The trick is algebra on the RLHF objective. Maximize expected reward minus β times the KL divergence from a reference policy, and the optimal policy has a closed form: π*(y|x) = π_ref(y|x) * exp(r(x, y) / β) / Z(x). Rearrange for the reward and you get r(x, y) = β log(π*(y|x) / π_ref(y|x)) + β log Z(x).
Plug that into the reward model’s pairwise loss. The Z(x) term is the same for both answers, so it cancels in the difference. What’s left is a loss written only in terms of the policy and the reference:
L_DPO = -log σ( β [ log(π(y_w|x) / π_ref(y_w|x)) - log(π(y_l|x) / π_ref(y_l|x)) ] )
In words: push up the log-probability ratio of the chosen answer relative to the reference, push down the rejected one’s, and stop pushing once the margin is large. The paper’s appendix gives it as a short PyTorch function built around losses = -F.logsigmoid(beta * (pi_logratios - ref_logratios)), and uses β = 0.1 unless noted otherwise. Hugging Face’s TRL DPO trainer defaults to the same β.
Does DPO actually match the RLHF optimum?
The derivation says it should. I wanted to see it, so here’s a test small enough to solve exactly: one prompt, five possible answers, a hidden “true” reward, and an SFT policy. I generate 5,000 preference pairs from that reward, train DPO on the pairs only, and compare the result with the RLHF optimum computed from the true reward.
# dpo.py: DPO on a tabular policy, checked against the RLHF optimum.
# Python 3 standard library only.
import math, random
random.seed(7)
sigmoid = lambda x: 1 / (1 + math.exp(-x))
names = ["concise, correct", "thorough, correct", "partly right", "wrong", "off-topic"]
r_true = [1.5, 1.3, 0.4, -1.0, -2.0] # hidden human reward, never shown to DPO
pi_ref = [0.20, 0.20, 0.30, 0.20, 0.10] # the SFT model's probabilities
beta = 0.5
# 1. Preference pairs: sample two answers from the SFT model, a "labeler" picks one.
pairs = []
while len(pairs) < 5000:
a, b = random.choices(range(5), weights=pi_ref, k=2)
if a == b:
continue
pairs.append((a, b) if random.random() < sigmoid(r_true[a] - r_true[b]) else (b, a))
# 2. DPO: loss = -log sigmoid(beta * [(log pi(w) - log ref(w)) - (log pi(l) - log ref(l))]).
# No reward model, no sampling, just a classification loss on the pairs.
theta = [math.log(p) for p in pi_ref] # start the policy at the SFT model
for step in range(400):
g = [0.0] * 5
for w, l in pairs:
margin = beta * ((theta[w] - math.log(pi_ref[w])) - (theta[l] - math.log(pi_ref[l])))
coef = beta * (1 - sigmoid(margin))
g[w] += coef; g[l] -= coef
theta = [t + 2.0 * gi / len(pairs) for t, gi in zip(theta, g)]
z = sum(math.exp(t) for t in theta)
pi_dpo = [math.exp(t) / z for t in theta]
# 3. The RLHF optimum with the true reward: pi*(y) = pi_ref(y) * exp(r(y) / beta) / Z.
un = [p * math.exp(r / beta) for p, r in zip(pi_ref, r_true)]
pi_star = [u / sum(un) for u in un]
# 4. DPO's implicit reward: beta * log(pi / pi_ref), compared after centering.
imp = [beta * math.log(p / q) for p, q in zip(pi_dpo, pi_ref)]
c_imp, c_true = sum(imp) / 5, sum(r_true) / 5
print(f"{'answer':<18}{'SFT':>6}{'DPO':>7}{'RLHF opt':>9}{'implicit r':>12}{'true r':>8}")
for i, n in enumerate(names):
print(f"{n:<18}{pi_ref[i]:>6.2f}{pi_dpo[i]:>7.2f}{pi_star[i]:>9.2f}"
f"{imp[i] - c_imp:>+12.2f}{r_true[i] - c_true:>+8.2f}")
answer SFT DPO RLHF opt implicit r true r
concise, correct 0.20 0.53 0.54 +1.49 +1.46
thorough, correct 0.20 0.38 0.36 +1.32 +1.26
partly right 0.30 0.09 0.09 +0.42 +0.36
wrong 0.20 0.00 0.00 -1.02 -1.04
off-topic 0.10 0.00 0.00 -2.20 -2.04
DPO never saw a reward value, only which answer won each pair. It still lands within two points of the RLHF optimum on every answer. The last two columns are the paper’s title in numbers: β log(π / π_ref) read off the trained policy recovers the hidden reward to within about 0.15, up to a constant. The language model really is a reward model.
The toy flatters DPO in one way: a tabular policy can represent any distribution, and the pairs cover every answer. A real LLM sees a tiny slice of possible answers, and that’s where the two methods start to differ.
Is DPO better than RLHF?
The DPO paper reports that DPO “exceeds PPO-based RLHF in ability to control sentiment of generations” and “matches or improves response quality in summarization and single-turn dialogue”. Production teams moved too. Meta’s Llama 3 paper says: “We also explored on-policy algorithms such as PPO, but found that DPO required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval.” Allen AI’s Tulu 3 uses SFT, then DPO, then RL with verifiable rewards.
But Llama 3 didn’t use plain offline DPO either. Three details from the paper are worth copying:
- Fresh data each round. They trained on “the most recent batches of preference data collected using the best performing models from the previous alignment rounds”, so the data matches the policy being trained. That moves DPO closer to on-policy.
- Mask formatting tokens. Header and end tokens appear in both chosen and rejected answers, and the contrastive loss then tries to raise and lower the same tokens at once. They saw “tail repetition or abruptly generating termination tokens” until they masked them.
- Add an NLL term. A negative log-likelihood loss on chosen answers, weighted 0.2, to stop the log probability of chosen answers from falling. They used β = 0.1 and a learning rate of 1e-5.
The second and third points show DPO’s characteristic failure: the loss only cares about the margin between chosen and rejected, so it can widen the gap by pushing both down. The TRL docs say the same thing plainly: the margin is “typically achieved by suppressing the likelihood of dispreferred completions rather than by increasing the likelihood of preferred ones.”
When does RLHF still win?
When the policy needs to learn from its own new samples. DPO learns from a fixed set of pairs; it can’t explore answers nobody wrote down. Online RL samples from the current model, scores those samples, and improves on them. For tasks where you can score any answer automatically, such as math with a checkable result or code with tests, that loop is where reasoning models came from. That’s RLVR, and it uses PPO or GRPO-style RL, not DPO. Tulu 3 uses both: DPO for general preferences, then RLVR for verifiable skills.
What I would do
For a team adapting an open-weight model to a product’s tone or policy, I’d pick DPO, after SFT, every time. It runs with ordinary fine-tuning infrastructure, pairs well with LoRA adapters, needs only two models in memory, and has one main knob. I’d start at β = 0.1, since both the DPO paper and Llama 3 used it, and tune from there.
Before training, I’d put most of the effort into the pairs: rejected answers that are plausibly wrong rather than obviously bad, chosen answers that match the product’s real standard, and pairs generated by a model close to the one I’m training. I’d track the log probability of chosen answers during training (TRL logs it as logps/chosen) and add the NLL term if it drops. And I’d only reach for online RL when I have a reward I can compute and trust, which in practice means a verifier, not a learned preference model.
For the bigger picture of where this sits, see how LLMs are trained.
FAQ
Does DPO need a reward model?
No. DPO trains the language model directly on preference pairs. The policy’s log-probability ratio against the reference model acts as an implicit reward, so no separate reward model is trained.
What does beta do in DPO?
It plays the same role as the KL coefficient in RLHF: it controls how far the policy may move from the reference model. Lower β allows larger moves; the DPO paper and Llama 3 both used 0.1.
Is DPO reinforcement learning?
Not in the usual sense. It doesn’t sample from the model or use a reward signal during training; it’s a supervised loss on a fixed dataset. It’s derived from the same KL-regularized objective RLHF optimizes, which is why the two target the same optimum.
Can I use DPO with LoRA?
Yes. DPO only needs log probabilities from the policy and a frozen reference, so it works with parameter-efficient fine-tuning. TRL’s DPOTrainer takes a peft_config such as LoraConfig(), and if you pass no reference model it uses the model’s state from before DPO training. It can also precompute reference log probabilities so the reference doesn’t stay in memory.
Sources
- Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, May 29, 2023
- Ouyang et al., Training language models to follow instructions with human feedback, March 4, 2022
- Llama Team, Meta, The Llama 3 Herd of Models, July 31, 2024, section 4.1.4
- Hugging Face, TRL DPO Trainer documentation (read 2026-10-31)
- Lambert et al., Tulu 3: Pushing Frontiers in Open Language Model Post-Training, November 22, 2024