Fine-Tune LLMs with Direct Preference Optimization (2026)

Direct Preference Optimization (DPO) lets you fine-tune large language models not by telling it what’s good, but by showing it what’s better, bypassing the complex reward model training entirely.

Let’s see DPO in action, not with abstract concepts, but with a concrete example. Imagine we have a base LLM, say llama-2-7b-chat-hf, and we want it to be better at summarizing news articles. Our goal is to make its summaries more concise and informative.

Here’s a simplified look at the data you’d prepare for DPO. It’s a list of prompts, and for each prompt, you have a "chosen" response and a "rejected" response.

[
  {
    "prompt": "Summarize this article: [Long news article text...]",
    "chosen": "[Concise and informative summary A]",
    "rejected": "[Longer, less informative summary B]"
  },
  {
    "prompt": "Summarize this article: [Another long news article text...]",
    "chosen": "[Concise and informative summary C]",
    "rejected": "[Vague summary D]"
  }
  // ... thousands more examples
]

The chosen response is what we prefer for that prompt, and rejected is what we don’t. DPO uses this pairwise comparison directly.

Now, how does this work under the hood? DPO is essentially a clever reparameterization of reinforcement learning from human feedback (RLHF). Traditional RLHF involves three steps:

Supervised Fine-Tuning (SFT): Train the base LLM on high-quality instruction-response pairs.
Reward Model (RM) Training: Train a separate model to predict a scalar reward score for a given prompt-response pair, based on human preferences.
RL Fine-Tuning: Use the trained RM as a reward function to fine-tune the SFT model using RL algorithms like Proximal Policy Optimization (PPO).

DPO collapses steps 2 and 3. It directly optimizes the LLM using the preference data. The core idea is to derive a loss function that encourages the LLM to assign a higher probability to the chosen response than to the rejected response, while also staying close to the original SFT model to avoid catastrophic forgetting or generating gibberish.

The DPO loss function looks something like this:

L(θ) = -log(σ(β * log(P_θ(y_c|x) / P_π(y_c|x)) / β) - log(σ(β * log(P_θ(y_r|x) / P_π(y_r|x)) / β)))

Where:

θ are the parameters of the LLM we are fine-tuning (the policy).
π are the parameters of the reference model (usually the initial SFT model, kept frozen).
x is the prompt.
y_c is the chosen response.
y_r is the rejected response.
P_θ(y|x) is the probability of generating response y given prompt x under the current policy θ.
P_π(y|x) is the probability of generating response y given prompt x under the reference policy π.
β is a hyperparameter controlling the strength of the preference signal (often set to 1.0).
σ is the sigmoid function.

This loss function, when minimized, directly increases the log-probability ratio of the chosen response over the rejected response, relative to the reference model. It’s like saying, "Make the 'good' answer more likely than the 'bad' answer, but don’t stray too far from what you already know."

The levers you control are primarily:

The dataset: The quality and quantity of your preference pairs are paramount. More diverse and representative preferences lead to a better-tuned model.
The reference model: This is typically your SFT model. It acts as a stable anchor.
The β hyperparameter: This balances the DPO objective with the KL divergence penalty (implicitly handled by comparing to the reference model). A higher β pushes the model more aggressively towards the preferences.
Training hyperparameters: Standard learning rate, batch size, and number of training epochs apply.

One thing that often surprises people is how sensitive the performance can be to the difference in quality between the chosen and rejected responses, rather than just the absolute quality. The model learns to distinguish subtle improvements. If your rejected responses are also quite good, the model has a harder time finding a clear signal to optimize for. Conversely, if your chosen responses are clearly superior and rejected are clearly inferior, the model learns the desired behavior much faster. It’s not about making responses "good," but about making them better than the alternative provided.

After fine-tuning with DPO, you might find that the model is now excellent at following instructions but sometimes hallucinates facts it wasn’t trained on.