# Hugging Face — Preference Tuning LLMs with Direct Preference Optimization Methods

- Company: Hugging Face (huggingface.co)
- Announced: 2024-01-18T00:00:00+00:00
- Category: research-paper
- Subject: Platform
- Models affected: OpenHermes-2.5-Mistral-7B, Zephyr-7b-beta-sft
- Source: https://huggingface.co/blog/pref-tuning
- Record: https://forck.live/items/1957-preference-tuning-llms-with-direct-preference-optimization-methods

This blog post empirically evaluates three preference tuning methods (DPO, IPO, KTO) on two 7B LLMs, sweeping hyperparameters and evaluating with MT-Bench.

## Evidence

Verbatim from https://huggingface.co/blog/pref-tuning:

> We evaluate three promising methods to align language models without reinforcement learning (or preference tuning) on a number of models and hyperparameter settings. In particular we train using different hyperparameters and evaluate on:
> 
> Direct Preference Optimization
> (DPO)
> 
> Identity Preference Optimisation
> (IPO)
> 
> Kahneman-Tversky Optimisation
> (KTO)

---

Record: https://forck.live/items/1957-preference-tuning-llms-with-direct-preference-optimization-methods
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md