Seonglae ChoPRM
Per-Token Rewards
Per-Chunk Rewards
CRL: Control Reinforcement Learning
openreview.net
https://openreview.net/pdf?id=jiPrwmMb2e
Dataset
In reasoning training, data distribution may be more important than correctness. SFT with incorrect CoT generated by a stronger model can outperform human-written perfectly correct CoT. The reasons are distribution proximity: synthetic CoT is closer to the student model's output distribution, making it easier to learn, and partial validity: incorrect CoT still contains many correct intermediate reasoning steps.
- Paraphrasing human CoT with LLM improves performance → proves distribution effect
- Gradually increasing error rate leads to gradual performance decline → demonstrates error tolerance
In other words, final answer accuracy is not a reliable indicator of CoT quality. In reasoning SFT data curation, "distribution alignment" is a more critical variable than correctness
arxiv.org
https://arxiv.org/pdf/2512.22255