When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO

Yu Li; Tian Lan; Zhengling Qi

cs.AIMar 13, 2026

When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO

Authors: Yu Li, Tian Lan, Zhengling Qi

Organizations: Department of Electrical and Computer Engineering, George Washington University · School of Business, George Washington University

Abstract

Group Relative Policy Optimization (GRPO) has emerged as an effective method for training reasoning models. While it computes advantages based on group mean, GRPO treats each output as an independent sample during the optimization and overlooks a vital structural signal: the natural contrast between correct and incorrect solutions within the same group, thus ignoring the rich, comparative data that could be leveraged by explicitly pitting successful reasoning traces against failed ones. To capitalize on this, we present a contrastive reformulation of GRPO, showing that the GRPO objective implicitly maximizes the margin between the policy ratios of correct and incorrect samples. Building on this insight, we propose Bilateral Context Conditioning (BICC), a mechanism that allows the model to cross-reference successful and failed reasoning traces during the optimization, enabling a direct information flow across samples. We further introduce Reward-Confidence Correction (RCC) to stabilize training by dynamically adjusts the advantage baseline in GRPO using reward-confidence covariance derived from the first-order approximation of the variance-minimizing estimator. Both mechanisms require no additional sampling or auxiliary models and can be adapted to all GRPO variants. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements across comprehensive models and algorithms. Code is available at https://github.com/Skylanding/BiCC.

Figures & tables

Figure 1: Overview of the proposed methods. The standard GRPO pipeline processes query $q$ through the policy model to generate $G$ outputs, which are scored by reward model and normalized to compute advantages for policy gradient update. BiCC partitions outputs by reward and conditions each sample on opposite-partition outputs, enabling cross-partition information flow where right and wrong attempts inform each other. RCC computes covariance between rewards and log-probability shifts to correct the advantage estimation.

Ratio $\rho_{i}$	Objective $\mathcal{L}_{i}$
GRPO ( Guo et al., 2025 )
$\dfrac{\pi_{\theta}(o_{i}\|q)}{\pi_{\theta_{\text{old}}}(o_{i}\|q)}$	$\min\bigl(\rho_{i}A_{i},\text{clip}(\rho_{i},1{\pm}\epsilon)A_{i}\bigr)$
$\dfrac{\pi_{\theta}(o_{i}\|{\color[rgb]{0,0.7,0}q,\mathcal{O}^{\mp}_{i}})}{\pi_{\theta_{\text{old}}}(o_{i}\|q)}$	$\min\bigl(\rho_{i}A_{i},\text{clip}(\rho_{i},1{\pm}\epsilon)A_{i}\bigr)$
Dr.GRPO ( Liu et al., 2025 )
$\dfrac{\pi_{\theta}(o_{i}\|q)}{\pi_{\text{ref}}(o_{i}\|q)}$	$\min\bigl(\rho_{i}A_{i},\text{clip}(\rho_{i},1{\pm}\epsilon)A_{i}\bigr)-\beta\mathbb{D}_{{\color[rgb]{0,0.7,0}\text{KL}}}$
$\dfrac{\pi_{\theta}(o_{i}\|{\color[rgb]{0,0.7,0}q,\mathcal{O}^{\mp}_{i}})}{\pi_{\text{ref}}(o_{i}\|q)}$	$\min\bigl(\rho_{i}A_{i},\text{clip}(\rho_{i},1{\pm}\epsilon)A_{i}\bigr)-\beta\mathbb{D}_{{\color[rgb]{0,0.7,0}\text{KL}}}$

Table 1: BiCC applied to GRPO variants. For each method, the first row shows the standard formulation and the shaded row shows the bilateral modification. Green highlights the change.

Figure 2: Reward-confidence correlation analysis. $\text{Cov}(R,\delta)$ increases throughout training for both models. Distribution of $\delta$ at training steps 2.5k–3k shows clear separation by reward, with mean separation $\Delta_{\mu}=0.27$ for Qwen3-4B and $0.56$ for Phi-4-mini.

	Qwen3-4B-Instruct-2507				Phi-4-mini-instruct-3.8B
Method	Math500	AMC 23	AIME24	AIME25	Math500	AMC 23	AIME24	AIME25
SFT Baseline	89.2	69.4	53.8	45.4	72.3	36.5	9.8	4.3
Standard GRPO Baseline
GRPO ( $G=2$ )	90.6	70.2	53.5	45.8	75.8	39.2	10.2	4.5
BiCC GRPO	91.0 $\uparrow$ 0.4	71.4 $\uparrow$ 1.2	54.2 $\uparrow$ 0.7	46.2 $\uparrow$ 0.4	76.4 $\uparrow$ 0.6	40.5 $\uparrow$ 1.3	10.5 $\uparrow$ 0.3	4.8 $\uparrow$ 0.3
GRPO ( $G=8$ )	91.4	72.8	54.0	46.4	76.2	42.5	10.4	4.5

Table 2: Main results on mathematical reasoning benchmarks. We report the average Pass@1 accuracy (%) over 32 independent runs using Qwen3-4B and Phi-4-mini. BiCC denotes our Bi lateral C onditioning extension applied to various GRPO-based baselines. Shaded rows represent the proposed variants, with subscripts indicating the absolute performance change relative to the corresponding base method. BiCC yields consistent gain of 0.3–1.9 percentage points across settings, with larger gains on weaker base models.

Panel of Figure 3 — Figure 3: Training and evaluation performance.

Context Ratio	Qwen3-4B	Phi-4-mini
20%	91.8	77.4
40%	92.2	78.1
60%	92.0	77.8

Table 3: Effect of context length allocation for opposite-partition samples on Math500 accuracy.

Figure 4: Evolution of

\delta

distributions during training. The separation between

R=0

and

R=1

partitions widens as training progresses, reflecting the model’s increasing ability to assign higher probability to correct outputs.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimization
Learning rate	$1\times 10^{-6}$
Optimizer (Adam $\beta_{1},\beta_{2}$ )	AdamW (0.9, 0.999)
Weight decay / Gradient clipping	0.01 / 1.0
LR schedule (warmup / total steps)	Cosine (100 / 3000)
Batch & Model

Appendix

Table 4: Hyperparameter configuration (shared across both models unless noted).

Method	Parameter	Value
Dr.GRPO	KL coefficient $\beta$ ; Reference policy	0.01; Frozen initial
DAPO	Clip bounds $(\epsilon_{l},\epsilon_{h})$	(0.2, 0.28)
GMPO	Geometric mean exponent	$1/T_{i}$
GSPO	Aggregation; Stop-gradient	Geometric mean; Yes

Appendix

Table 5: Variant-specific hyperparameters.

Benchmark	Problems	Difficulty	Answer Type	Source
Math500	500	Competition	Integer/Expr	Hendrycks et al. (2021)
AMC 2023	40	Competition	Integer	AMC 10/12
AIME 2024	30	Olympiad	Integer (0–999)	AIME I/II
AIME 2025	30	Olympiad	Integer (0–999)	AIME I/II
Evaluation Configuration : Temperature 0.6, Top- $p$ 0.95, Max tokens 4096, Seeds 0–31

Appendix

Table 6: Evaluation benchmarks and sampling configuration.

Configuration	Qwen3-4B	Phi-4-mini
GRPO (baseline)	91.4	76.2
BICC-GRPO	92.2	78.1
BICC-GRPO + RCC	92.6	78.8

Appendix

Table 7: Ablation of BICC and RCC on Math500 (%).

Model	0–0.5k	0.5k–1k	1k–1.5k	1.5k–2k	2k–2.5k	2.5k–3k
Qwen3-4B	0.015	0.029	0.043	0.054	0.063	0.066
Phi-4-mini	0.037	0.062	0.090	0.113	0.131	0.138

Appendix

Table 8:

\text{Cov}(R,\delta)

across training stages.

Explore similar work

VeriGate: Verifier-Gated Step-Level Supervision for GRPO
May 28, 2026Aakriti Agrawal, Minghui Liu, Furong HuangProcess Reward Models RL for Language Model Reasoning
University of Maryland, College Park
ReCo: Reweighting GRPO Against Distributional Concentration
Jul 29, 2026Junoh Park, Junseo Hwang, Wonguk Cho +1Reasoning Diversity RL for Language Model Reasoning
Graduate School of Data Science, Seoul National University
Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
Jul 21, 2026Priyank Agrawal, Ankur Samanta, Shervin Ghasemlou +4Reinforcement Learning RL for Language Model Reasoning
1Meta AI · 2Columbia University · *Work done at Meta

May 28, 2026J/K move · Enter open · S save

VeriGate: Verifier-Gated Step-Level Supervision for GRPO

Aakriti Agrawal, Minghui Liu, Furong Huang

University of Maryland, College Park

Process Reward Models RL for Language Model Reasoning Group Relative Policy Optimization Credit Assignment in RL Reinforcement Learning with Verifiable Rewards Step-Level Credit Assignment

Open paper AskPDF arXiv

Group Relative Policy Optimization (GRPO) is an effective recipe for training reasoning models with verifier-based outcome rewards, but its supervision is sparse: when all sampled trajectories for a prompt receive the same verifier reward, the group-relative advantage collapses to zero and learning stalls. Outcome-only rewards also provide no step-level credit assignment, limiting exploration and making it harder to learn robust reasoning. We present VeriGate (Verifier-Gated Step-Level GRPO), a verifier-gated extension of GRPO that addresses these limitations with three design choices. First, VeriGate keeps the verifier in charge whenever verifier rewards induce a meaningful preference among sampled trajectories, and uses process supervision only when verifier rewards are degenerate. Second, instead of collapsing Process Reward Model (PRM) step scores into a single trajectory reward, VeriGate converts them into future-cumulated rewards to assign continuation-aware credit. Third, VeriGate transforms these rewards into group-normalized token-level advantages, restoring informative gradients and fine-grained credit assignment while remaining less susceptible to reward hacking than methods that optimize aggregated PRM scores. Empirically, training on MATH with 1.5B and 7B Qwen2.5-Instruct models and evaluating on six reasoning benchmarks, VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively, substantially reduces zero-gradient failures, decreases reward-hacking behavior, and improves reasoning quality relative to outcome-only GRPO and PRM-as-outcome baselines.

Abstract

Explore similar work

VeriGate: Verifier-Gated Step-Level Supervision for GRPO

ReCo: Reweighting GRPO Against Distributional Concentration

Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information