DPO (Direct Preference Optimization)

Definition

Direct Preference Optimization (DPO) is a fine-tuning method that aligns a language model with human preferences by optimizing directly on pairs of (chosen, rejected) responses. Unlike RLHF, DPO skips training a separate reward model and skips reinforcement learning entirely. Instead, it derives a loss function that a standard supervised training loop can optimize.

The key insight of Rafailov et al.'s paper — Direct Preference Optimization: Your Language Model is Secretly a Reward Model — is a mathematical reformulation. The RLHF objective (maximize expected reward subject to a KL constraint from a reference model) has a closed-form optimal policy in terms of the reward function. DPO inverts this relationship: instead of learning a reward and then optimizing, it treats the model itself as an implicit reward function and optimizes preferences directly.

In practice, DPO looks almost identical to supervised fine-tuning. You need pairs of preferred and dispreferred responses. The loss encourages the model to increase the probability of preferred responses relative to dispreferred ones, weighted by how much both differ from a reference model. Training is stable, cheap, and can be done with any standard LoRA or full fine-tuning pipeline.

The trade-offs vs RLHF are genuine but usually favorable. DPO is simpler to implement (no PPO), faster to train (no reward model, no rollouts), and more sample-efficient in many settings. RLHF retains advantages for very complex reward signals and scenarios where online sampling matters. Most teams today start with DPO and only reach for RLHF when specific needs emerge — see our comparison.

Real-world example

DPO in a few lines with HuggingFace TRL

A dataset for DPO consists of records like {prompt, chosen, rejected}. The chosen response is the one humans preferred; the rejected is the alternative. Given a base SFT model, DPO training updates the model so that the log-probability of chosen responses increases relative to rejected ones — while a KL term keeps the trained model close to the reference. In production this is often a few hundred lines of code and a few hours on a single GPU for LoRA-based DPO on a 7-8B model.

# DPO training with TRL — much simpler than RLHF
from datasets import load_dataset
from trl import DPOTrainer, DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig

# 1. Load SFT'd base model + reference (frozen)
model = AutoModelForCausalLM.from_pretrained("./sft-model")
ref_model = AutoModelForCausalLM.from_pretrained("./sft-model")
tokenizer = AutoTokenizer.from_pretrained("./sft-model")

# 2. Load preference dataset
# Format: {"prompt": "...", "chosen": "...", "rejected": "..."}
dataset = load_dataset("json", data_files="preferences.jsonl")

# 3. Configure DPO
config = DPOConfig(
    output_dir="./dpo-model",
    beta=0.1,                    # KL regularization strength
    learning_rate=5e-7,          # much lower than SFT
    per_device_train_batch_size=4,
    num_train_epochs=1,
)

# 4. Train (no PPO, no reward model!)
trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,
    args=config,
    train_dataset=dataset["train"],
    tokenizer=tokenizer,
    peft_config=LoraConfig(r=16, target_modules=["q_proj","v_proj"]),
)
trainer.train()
trainer.save_model("./dpo-model")

# That's it. No reward model. No rollouts. No PPO stability issues.

When you'll encounter this

• You're doing new preference training — DPO is the default starting choice in 2026.
• You have preference pairs (chosen, rejected) — the standard format for both DPO and RLHF.
• You want to avoid RL complexity — no PPO tuning, no reward hacking mitigation, no online rollouts.
• You have limited GPU budget — DPO is significantly cheaper than RLHF for equivalent results.
• You're fine-tuning open-weight models — DPO integrates cleanly with LoRA and works on a single GPU for 7-8B models.

How it works

Common misconceptions

Related terms

Where you'll see this in practice

Learn more

• Read the deep-dive: How DPO Works
• Compare RLHF vs DPO
• Take the Master Fine-Tuning path
• Browse Fine-tuning Methods category