Group Relative Policy Optimization (GRPO) has emerged as an effective method for training reasoning models. While it computes advantages based on group mean, GRPO treats each output as an independent sample during the optimization and overlooks a vital structural signal: the natural contrast between correct and incorrect solutions within the same group, thus ignoring the rich, comparative data that could be leveraged by explicitly pitting successful reasoning traces against failed ones. To capitalize on this, we present a contrastive reformulation of GRPO, showing that the GRPO objective implicitly maximizes the margin between the policy ratios of correct and incorrect samples. Building on this insight, we propose Bilateral Context Conditioning (BICC), a mechanism that allows the model to cross-reference successful and failed reasoning traces during the optimization, enabling a direct information flow across samples. We further introduce Reward-Confidence Correction (RCC) to stabilize training by dynamically adjusts the advantage baseline in GRPO using reward-confidence covariance derived from the first-order approximation of the variance-minimizing estimator. Both mechanisms require no additional sampling or auxiliary models and can be adapted to all GRPO variants. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements across comprehensive models and algorithms. Code is available at https://github.com/Skylanding/BiCC.
Figures & tables
Figure 1: Overview of the proposed methods. The standard GRPO pipeline processes query q through the policy model to generate G outputs, which are scored by reward model and normalized to compute advantages for policy gradient update. BiCC partitions outputs by reward and conditions each sample on opposite-partition outputs, enabling cross-partition information flow where right and wrong attempts inform each other. RCC computes covariance between rewards and log-probability shifts to correct the advantage estimation.
Table 1: BiCC applied to GRPO variants. For each method, the first row shows the standard formulation and the shaded row shows the bilateral modification. Green highlights the change.
Figure 2: Reward-confidence correlation analysis. Cov(R,δ) increases throughout training for both models. Distribution of δ at training steps 2.5k–3k shows clear separation by reward, with mean separation Δμ=0.27 for Qwen3-4B and 0.56 for Phi-4-mini.
Qwen3-4B-Instruct-2507
Phi-4-mini-instruct-3.8B
Method
Math500
AMC 23
AIME24
AIME25
Math500
AMC 23
AIME24
AIME25
SFT Baseline
89.2
69.4
53.8
45.4
72.3
36.5
9.8
4.3
Standard GRPO Baseline
GRPO ( G=2 )
90.6
70.2
53.5
45.8
75.8
39.2
10.2
4.5
BiCC GRPO
91.0 ↑ 0.4
71.4 ↑ 1.2
54.2 ↑ 0.7
46.2 ↑ 0.4
76.4 ↑ 0.6
40.5 ↑ 1.3
10.5 ↑ 0.3
4.8 ↑ 0.3
GRPO ( G=8 )
91.4
72.8
54.0
46.4
76.2
42.5
10.4
4.5
Table 2: Main results on mathematical reasoning benchmarks. We report the average Pass@1 accuracy (%) over 32 independent runs using Qwen3-4B and Phi-4-mini. BiCC denotes our Bi lateral C onditioning extension applied to various GRPO-based baselines. Shaded rows represent the proposed variants, with subscripts indicating the absolute performance change relative to the corresponding base method. BiCC yields consistent gain of 0.3–1.9 percentage points across settings, with larger gains on weaker base models.
Figure 3: Training and evaluation performance.
Context Ratio
Qwen3-4B
Phi-4-mini
20%
91.8
77.4
40%
92.2
78.1
60%
92.0
77.8
Table 3: Effect of context length allocation for opposite-partition samples on Math500 accuracy.
Figure 4: Evolution of δ distributions during training. The separation between R=0 and R=1 partitions widens as training progresses, reflecting the model’s increasing ability to assign higher probability to correct outputs.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Optimization
Learning rate
1×10−6
Optimizer (Adam β1,β2 )
AdamW (0.9, 0.999)
Weight decay / Gradient clipping
0.01 / 1.0
LR schedule (warmup / total steps)
Cosine (100 / 3000)
Batch & Model
Appendix
Table 4: Hyperparameter configuration (shared across both models unless noted).
Method
Parameter
Value
Dr.GRPO
KL coefficient β ; Reference policy
0.01; Frozen initial
DAPO
Clip bounds (ϵl,ϵh)
(0.2, 0.28)
GMPO
Geometric mean exponent
1/Ti
GSPO
Aggregation; Stop-gradient
Geometric mean; Yes
Appendix
Table 5: Variant-specific hyperparameters.
Benchmark
Problems
Difficulty
Answer Type
Source
Math500
500
Competition
Integer/Expr
Hendrycks et al. (2021)
AMC 2023
40
Competition
Integer
AMC 10/12
AIME 2024
30
Olympiad
Integer (0–999)
AIME I/II
AIME 2025
30
Olympiad
Integer (0–999)
AIME I/II
Evaluation Configuration : Temperature 0.6, Top- p 0.95, Max tokens 4096, Seeds 0–31
Appendix
Table 6: Evaluation benchmarks and sampling configuration.
Group Relative Policy Optimization (GRPO) is an effective recipe for training reasoning models with verifier-based outcome rewards, but its supervision is sparse: when all sampled trajectories for a prompt receive the same verifier reward, the group-relative advantage collapses to zero and learning stalls. Outcome-only rewards also provide no step-level credit assignment, limiting exploration and making it harder to learn robust reasoning. We present VeriGate (Verifier-Gated Step-Level GRPO), a verifier-gated extension of GRPO that addresses these limitations with three design choices. First, VeriGate keeps the verifier in charge whenever verifier rewards induce a meaningful preference among sampled trajectories, and uses process supervision only when verifier rewards are degenerate. Second, instead of collapsing Process Reward Model (PRM) step scores into a single trajectory reward, VeriGate converts them into future-cumulated rewards to assign continuation-aware credit. Third, VeriGate transforms these rewards into group-normalized token-level advantages, restoring informative gradients and fine-grained credit assignment while remaining less susceptible to reward hacking than methods that optimize aggregated PRM scores. Empirically, training on MATH with 1.5B and 7B Qwen2.5-Instruct models and evaluating on six reasoning benchmarks, VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively, substantially reduces zero-gradient failures, decreases reward-hacking behavior, and improves reasoning quality relative to outcome-only GRPO and PRM-as-outcome baselines.
Group Relative Policy Optimization (GRPO) has become a standard reinforcement learning method for post-training language models. Recent work shows that GRPO can reduce the base model's reasoning capacity and underperform it in Pass@k when k is large, indicating reduced coverage of reasoning paths. We find that this reduction is associated with GRPO concentrating on responses that the base model already generates with high probability. We trace this concentration to two mechanisms in the GRPO update. At the response level, high-probability responses dominate the group gradient through repeated occurrence. At the token level, GRPO's importance ratio scales gradients, further reinforcing tokens that become more likely under the current policy. We propose ReCo, a reweighting method that addresses both effects. Response contributions are normalized by their expected occurrence within the rollout group, and the token-level importance ratio is replaced with a variance-based ratio that gives larger update scale to non-saturated decision points where alternative token choices remain plausible. Across Qwen2.5-Math-1.5B/7B and Llama-3.1-8B-Instruct on five mathematical reasoning benchmarks, ReCo improves Pass@k for large values of k and is comparable to GRPO for small values of k.
Junoh Park, Junseo Hwang, Wonguk Cho +1
Graduate School of Data Science, Seoul National University
Reinforcement learning with verifiable rewards (RLVR) improves reasoning in large language models. Yet, typical RLVR approaches fail on difficult problems: when a model cannot generate any correct solutions, it receives zero learning signal. Providing privileged guidance during training, such as solution prefixes, can help overcome this learning cliff by steering the model towards {correct solutions with non-zero reward}. {We call these rollouts off-context: they are generated from a training prompt that contains privileged guidance, while the target objective is defined by the original prompt without that guidance.} {We introduce} Off-Context GRPO (OC-GRPO), a minimally modified variant of GRPO that uses guided rollouts but applies an importance-corrected objective to steer the update back toward the original unguided objective, avoiding the mismatch that destabilizes uncorrected guided training. Empirically, our algorithm achieves a 3.9% absolute improvement (13.8% relative gain) over vanilla GRPO on average across standard mathematical reasoning benchmarks with negligible additional cost.