DPO

Creator

Creator

Seonglae Cho

Created

Created

2023 Sep 24 4:20

Editor

Editor

Seonglae Cho

Edited

Edited

2024 Aug 8 2:19

Refs

Refs

Direct Preference Optimization

RL 없이도 Human Preference를 학습 RLHF에서

PPO architecture 보다 훨적은 memory사용

사용자 선호도 쌍의 집합과 logit으로 사용자의 pairwise preferences를 직접적으로 모델 학습에 반영

notion image

sDPO from upstage

Implementation

dpo-from-scratch.ipynb

towardsdatascience.com

https://towardsdatascience.com/fine-tune-a-mistral-7b-model-with-direct-preference-optimization-708042745aac

https://arxiv.org/pdf/2305.18290.pdf

Direct Preference Optimization 논문리뷰

Direct Preference Optimization: Your Language Model is Secretly a Reward Model 논문리뷰

Direct Preference Optimization 논문리뷰

https://junbuml.ee/dpo

Self-Rewarding Language Models

Paper page - Self-Rewarding Language Models

Join the discussion on this paper page

Paper page - Self-Rewarding Language Models

https://huggingface.co/papers/2401.10020

Paper page - Self-Rewarding Language Models

DPO Datasets

argilla/OpenHermesPreferences · Datasets at Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

argilla/OpenHermesPreferences · Datasets at Hugging Face

https://huggingface.co/datasets/argilla/OpenHermesPreferences

argilla/OpenHermesPreferences · Datasets at Hugging Face

Recommendations

///////