Seonglae ChoOPD
- Student가 자신의 Policy로 Rollout 생성
- Teacher가 해당 토큰들에 대한 확률분포 제공
- Student가 Teacher와의 KL Divergence를 최소화하도록 학습
핵심은 Base Model이 혼자서는 풀지 못하는 문제를, 전문가 정보를 이용해 학습시키면서 기존 능력은 보존하는 것
- Student: 문제를 풀다가 잘못된 방향으로 진행.
- Teacher: 정답 풀이를 추가로 제공받아 올바른 다음 토큰 분포를 생성.
- Distillation: Student가 자신의 잘못된 추론 경로에서도 올바른 방향으로 진행하도록 학습.
일반 SFT는 전문가가 작성한 정답 경로만 학습하지만, OPSD는 Student가 실제로 방문하는 상태에서 어떻게 행동해야 하는지 학습합니다.
- Distribution Shift 감소: 실제 추론 과정에서 방문하는 context로 학습.
- Catastrophic Forgetting 완화: 기존 모델의 행동 분포에서 크게 벗어나지 않는 업데이트.
- 새로운 지식 습득: Teacher만 접근할 수 있는 전문가 정보를 Student에게 전달.
엄밀히 답 풀이 제공받으면 on-policy가 아니지만 Student가 어떤 분포에서 학습용 trajectory를 생성했는지에 따라 On-policy라고 부르는 이유. 일반 SFT랑 유사하지만 좀더 섬세하게 distillation 하려는 시도
Rethinking-OPDThinking-Space • Updated 2026 Oct 26 6:51
Rethinking-OPD
Thinking-Space • Updated 2026 Oct 26 6:51
Built on GRPO, the work proposes two practical strategies to “rescue” OPD when it fails. (1) Off-policy cold start: before starting OPD, first run SFT on the student using the teacher’s rollout data to increase the initial overlap ratio and close the gap in thinking patterns. (2) Teacher-aligned prompt selection: use prompts that are similar to the teacher model’s post-training data to maximize compatibility between teacher and student.
It identifies two key conditions for OPD success: thinking-pattern consistency and knowledge novelty. It analyzes how, at the token level, the student and teacher distributions become gradually aligned along trajectories visited by the student, focusing on the shared overlap tokens between the two models’ top-k sets, . Quantitatively, the probability mass over overlap tokens (overlap mass), , accounts for 97%~99% of the total, showing that most of the OPD gradient signal arises from this region.
OPD loss is not a GRPO loss; it is reverse-KL distillation on student-generated trajectories. The paper’s standard OPD objective is not a GRPO surrogate but , i.e., a sequence-level reverse KL over student-generated trajectories. With autoregressive factorization it becomes , giving an “exact token-level decomposition,” where and . Implementation is divided into three supervision granularities: sampled-token OPD with and , which is an unbiased single-sample estimator since ; full-vocabulary OPD with ; and top-k OPD, minimizing over renormalized distributions on the student’s top-k set .
On-Policy Self-Training
별도의 외부 Teacher 대신, Base Model에 전문가 정답 등 privileged information을 제공해 Teacher로 활용하는 방식
