Texonom
Texonom/
Engineering
Engineering/Data Engineering/Artificial Intelligence/AI Risk/AI Alignment/Preference Optimization/
DPO
Search

DPO

Creator
Creator
Seonglae ChoSeonglae Cho
Created
Created
2023 Sep 24 4:20
Editor
Editor
Seonglae ChoSeonglae Cho
Edited
Edited
2025 Nov 7 11:23
Refs
Refs
RLHF
SFT
PPO
KTO

Direct Preference Optimization

Literally upweight the preferred response while downweight the unpreferred response, which is a really simple mechanism.
It can learn human preferences without RL, using significantly less memory compared to the
PPO architecture in RLHF.
The model directly incorporates user's pairwise preferences into training using a set of preference pairs and logits.
notion image
 
https://arxiv.org/pdf/2402.01306
 
 

Implementation

dpo-from-scratch.ipynb
rasbt
towardsdatascience.com
https://towardsdatascience.com/fine-tune-a-mistral-7b-model-with-direct-preference-optimization-708042745aac
arxiv.org
https://arxiv.org/pdf/2305.18290.pdf
Direct Preference Optimization 논문리뷰
Direct Preference Optimization: Your Language Model is Secretly a Reward Model 논문리뷰
Direct Preference Optimization 논문리뷰
https://junbuml.ee/dpo

Self-Rewarding Language Models

Paper page - Self-Rewarding Language Models
Join the discussion on this paper page
Paper page - Self-Rewarding Language Models
https://huggingface.co/papers/2401.10020
Paper page - Self-Rewarding Language Models

DPO Datasets

argilla/OpenHermesPreferences · Datasets at Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
argilla/OpenHermesPreferences · Datasets at Hugging Face
https://huggingface.co/datasets/argilla/OpenHermesPreferences
argilla/OpenHermesPreferences · Datasets at Hugging Face

sDPO from upstage

sDPO: Don’t Use Your Data All at Once
As development of large language models (LLM) progresses, aligning them with human preferences has become increasingly important. We propose stepwise DPO (sDPO), an extension of the recently popularized direct preference optimization (DPO) for alignment tuning. This approach involves dividing the available preference datasets and utilizing them in a stepwise manner, rather than employing it all at once. We demonstrate that this method facilitates the use of more precisely aligned reference models within the DPO training framework. Furthermore, sDPO trains the final model to be more performant, even outperforming other popular LLMs with more parameters.
sDPO: Don’t Use Your Data All at Once
https://arxiv.org/html/2403.19270v1

Toxicity reduction interpretation

DPO reduces toxicity not through a few neurons, but via distributed activation shifts across all MLP neurons. DPO operates through balanced action of four neuron groups:
  • TP↓: Toxicity-aligned + positive activation → decrease
  • TN↓: Toxicity-aligned + negative activation → decrease
  • AP↓: Anti-toxicity aligned + positive activation → increase (anti-toxicity reinforcement)
  • AN↓: Anti-toxicity aligned + negative activation → increase (anti-toxicity reinforcement)
Patching activations of all four groups to post-DPO values reproduces or exceeds DPO effects. In contrast, patching only toxic neurons has minimal effect.
arxiv.org
https://arxiv.org/pdf/2411.06424
suppresses a few toxic neurons
arxiv.org
https://arxiv.org/pdf/2401.01967
 
 

Backlinks

TorchTuneAutomated Behavior ElicitingContinuous-Adversarial TrainingSAE Finetuning TransferabilityPreference OptimizationMechanistic AI JailbreakFine Tuning

Recommendations

Texonom
Texonom/
Engineering
Engineering/Data Engineering/Artificial Intelligence/AI Risk/AI Alignment/Preference Optimization/
DPO