RLHF (Reinforcement Learning from Human Feedback)
At a glance
Definition
Reinforcement Learning from Human Feedback (RLHF) is a training technique that aligns a language model with human preferences by using human judgments to define a reward signal, then optimizing the model via reinforcement learning against that reward. It's what transforms a base language model — which just predicts the next token — into a helpful chat assistant that follows instructions, refuses harmful requests, and produces responses humans actually want.
The technique has three phases. First, supervised fine-tuning (SFT) teaches the base model to respond in a conversational format using demonstration data. Second, a reward model is trained on human preference data — pairs of responses where humans marked which was better. Third, Proximal Policy Optimization (PPO) (or a similar RL algorithm) optimizes the LLM to maximize the reward model's scores while staying close to the SFT model to avoid drifting into gibberish.
RLHF was introduced in Christiano et al.'s 2017 paper Deep Reinforcement Learning from Human Preferences, which showed the technique for simple robotic tasks. Its language model breakthrough came with OpenAI's 2022 InstructGPT paper — the technique that directly led to ChatGPT's launch later that year. Anthropic built Claude using RLHF and later evolved it into Constitutional AI.
By 2026, RLHF has both spread widely and been partly displaced. Every frontier chat model uses RLHF or a close variant. But newer alternatives — DPO skips the reward model entirely, GRPO from DeepSeek powers reasoning training, and RLAIF uses AI feedback instead of humans — offer simpler or more efficient alternatives. RLHF remains the foundational technique in the family.
Real-world example
How RLHF turned GPT-3 into ChatGPT
Before RLHF, GPT-3 could complete text but often responded to questions with more questions, refused to answer nothing, or produced unhelpful continuations. The InstructGPT paper demonstrated that a much smaller (1.3B parameter) RLHF-trained model was preferred over the raw 175B GPT-3 in human evaluations. Human labelers compared model outputs and marked preferences. A reward model learned from those preferences. PPO optimized the model to produce responses humans would rate highly. The result: a model that actually answered questions, followed instructions, and refused harmful requests. That model became the basis of ChatGPT — and started the current era of AI.
# RLHF with TRL (HuggingFace)
# High-level PPO training loop — real implementations are more complex.
from trl import PPOTrainer, PPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
# 1. Load SFT'd model and reward model
model = AutoModelForCausalLM.from_pretrained("./sft-model")
reward_model = AutoModelForCausalLM.from_pretrained("./reward-model")
ref_model = AutoModelForCausalLM.from_pretrained("./sft-model") # frozen ref
tokenizer = AutoTokenizer.from_pretrained("./sft-model")
# 2. Configure PPO
config = PPOConfig(
learning_rate=1.4e-5,
batch_size=64,
mini_batch_size=8,
ppo_epochs=4,
kl_penalty="kl", # KL to ref model — prevents drift
init_kl_coef=0.2,
)
trainer = PPOTrainer(
config=config, model=model, ref_model=ref_model, tokenizer=tokenizer,
)
# 3. Training loop
for batch in prompt_dataloader:
# Model generates responses
responses = trainer.generate(batch["input_ids"])
# Reward model scores them
rewards = reward_model_score(responses)
# PPO updates model to maximize rewards
# while penalizing divergence from ref model
stats = trainer.step(batch["input_ids"], responses, rewards)
trainer.save_pretrained("./rlhf-model")When you'll encounter this
- You're training a chat model from a base LLM — RLHF is the classic technique for teaching helpfulness.
- You need to align model outputs with specific human preferences — tone, style, safety, refusal behavior.
- You're optimizing for something you can't easily specify with a loss function — 'be helpful,' 'be safe,' 'be honest.'
- You have substantial human labeling budget — RLHF requires thousands to tens of thousands of preference labels.
- You need battle-tested alignment technology — RLHF has years of production use; alternatives are newer.
How it works
Phase 1: Supervised fine-tuning
Fine-tune the base LLM on human-written demonstrations of ideal responses. This gets the model responding in the right format and tone — the foundation for later RL work.
Collect preference data
Sample multiple responses from the SFT model for the same prompt. Human labelers compare pairs and mark which is better. Thousands to hundreds of thousands of comparisons build the dataset.
Train reward model
A separate model (usually the SFT model with a new head) is trained to predict which of two responses a human would prefer. The reward model outputs a scalar score for any response.
Set up PPO training
The LLM is now the policy. Its actions are token generations. Rewards come from the reward model. A frozen copy of the SFT model serves as a reference to prevent drift.
KL penalty
Add a penalty to keep the trained model close to the reference. Without this, the model quickly learns to exploit the reward model — producing high-reward gibberish. The KL coefficient is a critical hyperparameter.
PPO optimization
PPO updates the policy to maximize expected reward while respecting the KL constraint. It generates rollouts, computes advantages, and applies clipped policy gradient updates.
Iterate and evaluate
Monitor for reward hacking, evaluate on held-out prompts, and often collect more preference data. RLHF is an iterative process, not a one-shot job.
Deploy
The RLHF-tuned model is what users interact with. It's demonstrably better at following instructions, refusing harmful requests, and producing preferred responses.
Common misconceptions
RLHF makes models sound preferred by human labelers. That's often correlated with honesty, but also with sounding confident, avoiding hedging, and producing polished prose — sometimes at the cost of accuracy. Sycophancy is a well-documented side effect.
PPO is the classic algorithm but not required. DPO skips PPO entirely by optimizing directly on preference pairs. IPO, KTO, and other variants provide alternatives. In 2026, DPO is more common than PPO for new work — simpler and often comparably effective.
The opposite. Reward model quality depends entirely on preference data quality. Bad labels → bad reward → bad RLHF. Labeler training, disagreement resolution, and continuous data quality checks are more important than algorithm choice.
Related terms
Where you'll see this in practice
Learn more
- Read the deep-dive: How RLHF Works
- Compare RLHF vs DPO
- Take the Master Fine-Tuning path
- Browse Fine-tuning Methods category
Sources & further reading
- Deep Reinforcement Learning from Human Preferences — Christiano et al., 2017 (original RLHF paper). Read →
- Training language models to follow instructions with human feedback — Ouyang et al., 2022 (InstructGPT). Read →
- Constitutional AI: Harmlessness from AI Feedback — Bai et al., 2022 (Anthropic). Read →
- Proximal Policy Optimization Algorithms — Schulman et al., 2017 (PPO). Read →
Frequently asked about RLHF (Reinforcement Learning from Human Feedback)
Increasingly, yes — for new work. DPO is simpler (no reward model, no PPO), cheaper, and often as effective. But RLHF has years of production hardening at frontier labs. Many teams still use RLHF for its flexibility on complex reward signals. See our comparison.
Frontier RLHF uses hundreds of thousands to millions of preference labels. Practical smaller RLHF projects work with 10K-100K labels. Below that, DPO's efficiency starts winning. Quality matters more than quantity — well-trained labelers with clear guidelines outperform larger but noisier datasets.
When the model learns to game the reward model rather than actually improve. Classic examples: producing overly long responses because the reward model prefers length, or being sycophantic because agreeing with the user gets higher scores. Mitigations include KL penalty, better reward model training, and iterative labeling.
Yes. LoRA-based RLHF on 7B-8B open-weight models is affordable — you need enough VRAM for the base model, reward model, and reference model. Tools like TRL and OpenRLHF make it accessible. Most teams start with DPO on smaller models today.
RLHF uses human labelers to compare responses. RLAIF uses an AI (often a strong LLM) to compare responses. RLAIF is dramatically cheaper and scales further, but quality depends on the AI labeler. Constitutional AI is a hybrid — AI feedback guided by human-written principles.