Seonglae ChoRLVR
Programmatic Reward, Verifiable Feedback
Externally provided feedback that can be objectively validated against specific criteria.
Humans do not consider all paths in parallel, nor do they give bonus points to every step of the process just because the answer was correct. They should regret intermediate steps that were strange and give extra points to those that were helpful. In other words, a reflection process that synthesizes multiple trials is also necessary.
- ORM (Outcome Reward Model): A method that gives rewards based only on the final output.
- Process Reward Model (Process Reward Model): A method that evaluates and rewards each step.
For Reasoning Data with GRPO
Rule-based Verifiable Reward. LLM itself is policy network.
- Accuracy rewards: Checking if the model’s final answer is correct
- Format rewards: Incentivizing a structured chain-of-thought (enclose CoT tokens by
<think>and</think>)
Verifiable Rewards
Deepseek R1
arxiv.org
https://arxiv.org/pdf/2501.12948
From Zero to Reasoning Hero: How DeepSeek-R1 Leverages Reinforcement Learning to Master Complex Reasoning
A Blog post by Yihua Zhang on Hugging Face
https://huggingface.co/blog/NormalUhr/deepseek-r1-explained
