GRPO

Creator
Creator
Seonglae Cho
Created
Created
2025 Jan 26 19:49
Editor
Edited
Edited
2025 Jul 10 20:29

Group Relative Policy Optimization

In conventional
PPO
, a separate critic network (value function estimator) is used to calculate the advantage. In contrast, GRPO does not use a separate critic network by grouping multiple responses to the same question and using intra-group reward statistics to compute relative advantage.
notion image

Multi policy-group based update

Instead of using PPO's fixed clipping range, multiple outputs for the same question are grouped together to calculate the relative performance (advantage) of each sample based on the performance distribution within the group. A group is formed by processing the same input (prompt) to generate multiple outputs, which are then grouped together for relative advantage calculation. Specifically, GRPO calculates the normalized advantage of each sample using the reward distribution from the reward model for multiple output groups for the same question without value model.
Using this calculated relative advantage, dynamic clipping is applied to reflect how much better or worse each output is compared to the group average instead of using
GAE
.
As a result, this improves learning stability and efficiency by updating well-performing outputs more significantly while applying smaller updates to outputs near the average.

Reference Policy

AI Reward Hacking
or preserving the inherent quality of language models requires a trust zone, which is why a
KL Divergence
term is added to prevent significant changes. Here, the reference policy refers to the original LLM. The old policy is the policy learned in the step just before the current iteration, where rollout is performed to calculate success rates and weights.
All
RLHF
-like language model RL methods have this to prevent
AI Reward Hacking
.

GRPO Surrogate Loss & KL penalty

  • Length-normalized reward: rinorm=rioir_i^{\text{norm}} = \frac{r_i}{|o_i|}
  • Mean and std of normalized rewards: 1Nj=1Nrjnorm\frac{1}{N}\sum_{j=1}^N r_j^{\text{norm}}, σ=1Nj=1N(rjnormμ)2\sigma = \sqrt{\frac{1}{N}\sum_{j=1}^N \bigl(r_j^{\text{norm}} - \mu\bigr)^2}
  • Std-normalized advantage: Ai=rinormμσA_i = \frac{r_i^{\text{norm}} - \mu}{\sigma}
LGRPO(θ)=1Gi=1G1oit=1oimin ⁣[πθ(oi,tq,oi,<t)πold(oi,tq,oi,<t)A^i,t,  g(ϵ,A^i,t)]βDKL[πθπref]\begin{aligned} \mathcal{L}_{\mathrm{GRPO}}(\theta) &= \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min\!\Bigl[ \frac{\pi_{\theta}(o_{i,t}\mid q, o_{i,<t})} {\pi_{\mathrm{old}}(o_{i,t}\mid q, o_{i,<t})}\,\hat A_{i,t},\; g\bigl(\epsilon,\hat A_{i,t}\bigr) \Bigr] -\beta\,D_{\mathrm{KL}}\bigl[\pi_{\theta}\,\|\,\pi_{\mathrm{ref}}\bigr] \end{aligned}
notion image

Normalization terms

  • Standard deviation normalization - Adjusts update signals to have variance of 1 to improve "training stability", but when there are large rewards within a group, the same reward gets flattened more as it's divided by a larger standard deviation
  • Length Normalization - Divides by the token count of each response to equalize the policy gradient impact of answers with different lengths. While it serves a similar role to a
    Discount factor
    , it has the issue of over-reinforcing "short and accurate" answers while being relatively lenient on "long and incorrect" answers

Conclusion

Through grouping, instead of using PPO's "clipped surrogate objective", it directly optimizes a simple PG objective using group average baseline. In other words, while it's still
PPO
-like
On-policy
, it has more flexibility and removed dependency to train critic network (no need to learn the value function to use advantage). The key achievement is that by applying GRPO solely to math problems, there was an overall improvement in universal chain-of-thought reasoning.
  • Removing critic network
  • Dynamic clipping

Limitation

GRPO's length normalization leads to bias favoring short correct answers while being lenient on long incorrect answers (length bias), while std normalization results in bias towards extreme difficulty samples (difficulty bias). Dr. GRPO: Removes both normalization terms to address these biases, improving token efficiency.
Vanilla GRPO has bias to increase reasoning length with wrong answers
Vanilla GRPO has bias to increase reasoning length with wrong answers
In fact, GRPO is an objective designed to focus on making decoding variations robust within the same inference by relying on
Text Generation Temperature
, rather than directly providing
AI Incentive
to
AI Reasoning
itself. While training that relies on Group
Verifiable Reward
to achieve temperature robustness has improved universal reasoning CoT performance, further verification is needed to confirm whether it actually improved reasoning itself.
 
 
 
 
 
 
 

Implementation

Dr. GRPO without normalization

8x A100 GPUs for 27 hours → 7B model achieves 43.3% on AIME 2024 (Zero-RL SOTA)
  • Mean reward: rˉ=1Nj=1Nrj\bar r = \frac{1}{N}\sum_{j=1}^N r_j
  • Raw advantage: Ai=rirˉA_i = r_i - \bar r (simplified) or Ai=NN1(rirˉ)A_i =\frac{N}{N-1}(r_i - \bar r) (expectation 0 for small batch due to the leave one out baseline)
LDr.GRPO(θ)=1Gi=1Gt=1oimin ⁣[πθ(oi,tq,oi,<t)πold(oi,tq,oi,<t)A^i,t,  g(ϵ,A^i,t)]βDKL[πθπref]\begin{aligned} \mathcal{L}_{\mathrm{Dr.\,GRPO}}(\theta) &= \frac{1}{G} \sum_{i=1}^{G} \sum_{t=1}^{|o_i|} \min\!\Bigl[ \frac{\pi_{\theta}(o_{i,t}\mid q, o_{i,<t})} {\pi_{\mathrm{old}}(o_{i,t}\mid q, o_{i,<t})}\,\hat A_{i,t},\; g\bigl(\epsilon,\hat A_{i,t}\bigr) \Bigr] -\beta\,D_{\mathrm{KL}}\bigl[\pi_{\theta}\,\|\,\pi_{\mathrm{ref}}\bigr] \end{aligned}
8x A100 GPUs for 27 hours → 7B model achieves 43.3% on AIME 2024 (Zero-RL SOTA)
  • Group size: 8
  • Learning rate: 1e-6
  • Without KL term (β=0\beta = 0)
  • Temperature 1
DeGRPO (Decoupled Group Relative Policy Optimization) for cost optimization with short answer

RL with
Verifiable Reward

The loss combines binary verification rewards and KL regularization, reformulating it as a weighted contrastive loss using old policy samples. This means good samples receive high scores while poor samples receive low scores. The optimal policy πₙ can be explicitly expressed in terms of reference policy π₀, previous policy πₙ₋₁, and success probability statistics pₙ₋₁, establishing a functional relationship pₙ = h(pₙ₋₁) that converges to a fixed point p*. Furthermore, it can be mathematically proven that the fixed point p* is always greater than the initial success probability p₀, demonstrating that GRPO effectively increases success probability through iterations. Additionally, in actual parametric policies (e.g., gradient descent), if statistical and approximation errors are small, the success probability is guaranteed to remain near p*.

GRPO++ (to build DeepSWE, )

Drawing ideas from DAPO, Dr.GRPO, LOOP/RLOO and others, the following improvements were added:
  • Clip High (DAPO): Increase surrogate loss upper bound to enhance exploration
  • No KL Loss (DAPO)
  • No Reward Std, Length Normalization (Dr.GRPO)
  • Leave One Out: Reduce variance by removing one sample when estimating advantage (Loop/)
  • Compact Filtering: Mask trajectories when reaching max context/steps/timeout
  • No Entropy Loss: Remove entropy loss to prevent instability
 
 
 

Recommendations