Seonglae ChoLanguage Model RL Methods
Language Model Reward Benchmarks
Reinforcement Learning Transformers
Language Model RL Frameworks
SFT builds the parts. RL learns to combine them
From Reasoning Traces to Reusable Modules: Understanding...
Post-training pipelines that combine supervised fine-tuning (SFT) with reinforcement learning (RL) have emerged as the key recipe for transforming large language models (LLMs) into robust...
https://arxiv.org/abs/2606.18089
Era of Experience
storage.googleapis.com
https://storage.googleapis.com/deepmind-media/Era-of-Experience%20/The%20Era%20of%20Experience%20Paper.pdf
However, it has low Sample efficiency in larger samples and only finds better reasoning paths within its existing capacity which makes its total problem solving coverage smaller
www.arxiv.org
https://www.arxiv.org/pdf/2504.13837
Fine-tuning 20B LLMs with RLHF on a 24GB consumer GPU
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
https://huggingface.co/blog/trl-peft
How to Fine-Tune LLMs in 2024 with Hugging Face
In this blog post you will learn how to fine-tune LLMs using Hugging Face TRL, Transformers and Datasets in 2024. We will fine-tune a LLM on a text to SQL dataset.
https://www.philschmid.de/fine-tune-llms-in-2024-with-trl
Agent RL vulnerability
Search LLMs trained with agentic RL may appear safe, but can be easily jailbroken by manipulating the timing of the search step. The RL objective itself fails to suppress harmful queries, making "search first" behavior a critical vulnerability.
arxiv.org
https://arxiv.org/pdf/2510.17431
Information density (bits/sample) is very low in early training. Sample efficiency Supervised learning provides the correct answer for every token, always obtaining a lot of information, but RL only has high information when the probability of a correct answer is around 50%. In the early stages of RL, there are almost no correct samples, leading to extreme gradient variance.
Thinking through how pretraining vs RL learn
And implications for RLVR progress
https://www.dwarkesh.com/p/bits-per-sample
