RLHF

Reinforcement learning from human feedback

인간의 피드백을 기반으로 보상 함수를 학습하고 이를 통해 policy를 업데이트

Limitation

LM의 근본적인 문제인 Size, hallucination을 아직까지는 개선할 수는 없는 한계점

Scaling 이슈, 너무 복잡

LLaVA-RLHF

Visual Instruction Tuning

https://llava-rlhf.github.io/

Multi Adapter RL (MARL) - a single base model for everything

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

https://huggingface.co/docs/trl/multi_adapter_rl

RLHF란?

RLHF에 대해 알아보자

https://velog.io/@nellcome/RLHF란

Reinforcement learning from human feedback

In machine learning, reinforcement learning from human feedback (RLHF) or reinforcement learning from human preferences is a technique that trains a "reward model" directly from human feedback and uses the model as a reward function to optimize an agent's policy using reinforcement learning (RL) through an optimization algorithm like Proximal Policy Optimization.[1][2] The reward model is trained in advance to the policy being optimized to predict if a given output is good (high reward) or bad (low reward). RLHF can improve the robustness and exploration of RL agents, especially when the reward function is sparse or noisy.[3]

https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback

OOD generalisation is crucial given the wide range of real-world scenarios in which these models are being used, while output diversity refers to the model’s ability to generate varied outputs and is important for a variety of use cases

RLHF generalises better than SFT to new inputs, particularly as the distribution shift between train and test becomes larger. However, RLHF significantly reduces output diversity compared to SFT across a variety of measures, implying a tradeoff in current LLM fine-tuning methods between generalisation and diversity.

arxiv.org

https://arxiv.org/pdf/2310.06452

RLHF

Reinforcement learning from human feedback

Limitation

LLaVA-RLHF

Recommendations