Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards

Xuanchang Zhang; Tong Zhang

cs.LGOct 14, 2026

Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards

Authors: Xuanchang Zhang, Tong Zhang

Organizations: University of Illinois Urbana-Champaign

Abstract

Reinforcement learning with verifiable rewards often trains language models on responses generated by a different, possibly stale behavior policy. Correcting this mismatch with learner-to-behavior importance weights can introduce high variance for long responses. We introduce Gibbs-Gap Policy Optimization (GGPO), which minimizes a log-mean-exp contrast of KL-adjusted rewards among responses to the same prompt, without learner-to-behavior importance weights. We derive GGPO from KL-regularized policy optimization: on policy, its population loss is exactly the KL-regularized optimality gap, and under suitable coverage the learner's single-response expected reward is at least the behavior policy's best-of- $K$ reward, minus terms for the raw finite-group loss, regularization, and finite training data. We characterize reference refresh through exact Gibbs-target composition and a finite-data final-policy reward bound under local reference control and tokenwise behavior mismatch. This refresh is empirically important: in an ablation, fixed-reference GGPO scores $32.96$ on the five-benchmark problem-weighted average, versus $46.58$ with the default refresh schedule. Experiments on mathematical reasoning further show that GGPO outperforms GRPO and AGRO under training-inference mismatch and on fixed offline data, while maintaining its performance as controlled policy staleness grows.

Figures & tables

1:	Initialize $\pi_{\theta}\leftarrow\pi_{0}$ , $\pi_{\mathrm{ref}}\leftarrow\pi_{0}$ , and $u\leftarrow 0$ .
2:	while $u<U$ do
3:		Synchronize rollout parameters with $\pi_{\theta}$ ; freeze the resulting $\pi_{\mathrm{beh}}$ .
4:		Initialize the ordered collection $\mathcal{C}\leftarrow\varnothing$ .
5:		while $\|\mathcal{C}\|<NG_{\mathrm{mb}}$ do
6:			Take the next prompt batch from the shuffled dataloader of $\mathcal{D}$ .
7:			Sample $m$ responses per prompt under $\pi_{\mathrm{beh}}$ .

Algorithm 1 Gibbs-Gap Policy Optimization with periodic reference refresh

Model and method	MATH-500	AIME 2024	AIME 2025	OlympiadBench	Minerva Math	Avg.
Qwen3-1.7B-Base + GRPO	68.86 $\pm$ 0.32	9.03 $\pm$ 0.90	5.14 $\pm$ 0.63	32.13 $\pm$ 0.27	28.33 $\pm$ 0.40	42.64 $\pm$ 0.18
+ AGRO	70.68 $\pm$ 0.33	9.27 $\pm$ 1.15	6.56 $\pm$ 0.81	35.73 $\pm$ 0.30	29.30 $\pm$ 0.44	45.07 $\pm$ 0.19
+ GGPO	72.43 $\pm$ 0.29	11.11 $\pm$ 0.89	8.30 $\pm$ 0.68	37.11 $\pm$ 0.25	30.64 $\pm$ 0.36	46.58 $\pm$ 0.16
Qwen3-4B-Base + GRPO	80.71	16.88	11.88	46.63	40.29	55.51
+ AGRO	80.55	16.88	12.40	47.21	40.71	55.81
+ GGPO	81.83	16.88	14.38	48.33	41.85	56.98

Table 1: Comparison under the default training setting (

N=8

). Final-checkpoint mean@32 accuracy after 8,000 optimizer updates on five mathematical-reasoning benchmarks (%). The final column is the problem-weighted average over the five benchmarks, which gives every evaluated problem equal weight. For Qwen3-1.7B-Base, values are the mean

\pm

sample standard deviation across training seeds 42, 1234, and 2027. For Avg., these statistics are computed across the per-seed problem-weighted averages. Qwen3-4B-Base uses seed 42. Bold marks the best value in each model-size block.

Figure 1: Training dynamics under the default training setting (

N=8

) on Qwen3-1.7B-Base. From left to right: the token-mean K3 diagnostic of behavior–learner mismatch ( Schulman, 2020 ) (

\times 10^{-3}

), policy entropy, MATH-500 best@32 (%), and MATH-500 mean@32 (%), each against optimizer updates (GRPO orange, AGRO green, GGPO blue). K3 is computed on mixed-reward-filtered training minibatches. In the first two panels, faint lines are the logged series (one point per eight updates) and solid lines a nine-point moving average. At each reported update, curves summarize seeds 42, 1234, and 2027 by their mean, with shaded bands spanning one sample standard deviation across seeds; MATH-500 is evaluated every 160 optimizer updates.

Figure 2: Sampling budget under the default training setting on Qwen3-1.7B-Base. Left: cumulative generated training responses, in millions, against optimizer updates. Right: MATH-500 mean@32 accuracy against the same cumulative response count. Each curve ends at 8,000 optimizer updates (GRPO orange squares, AGRO green triangles, GGPO blue circles). Counts include responses discarded by mixed-reward filtering and generated during refill. GGPO finishes with the highest accuracy and the lowest response count.

Method	Fixed reference	$T_{\mathrm{ref}}=160$	$T_{\mathrm{ref}}=320$
GRPO	$42.64^{\dagger}$	42.40	42.48
AGRO	41.48	$45.07^{\dagger}$	43.86
GGPO	32.96	$46.58^{\dagger}$	44.96

Table 2: Reference refresh on Qwen3-1.7B-Base. Problem-weighted five-benchmark mean@32 (%) after 8,000 updates, with

\beta=0.01

. A dagger marks default settings reported as means over seeds 42, 1234, and 2027; other cells use seed 42. Table 6 gives the full results.

Figure 3: Controlled policy staleness on Qwen3-1.7B-Base. Problem-weighted final-checkpoint mean@32 accuracy over the five benchmarks after 8,000 optimizer updates, against synchronization delay

N

(equally spaced positions; values in Table 4 ). GRPO+TIS is evaluated at

N=128

. Results at

N=8,128

are three-seed means, with error bars showing

\pm 1

sample standard deviation;

N=16,32,64

use seed 42.

Figure 4: Offline training on Qwen3-1.7B-Base. Within each seed, all methods use the same fixed rollout pool, batch geometry, prompt order, and 400-update budget (GRPO orange, GRPO+TIS purple dashed, AGRO green, GGPO blue). From left: MATH-500 mean@32 (%), with the reported mean accuracy and bands spanning one reported sample standard deviation (Table 5 ); then, for seed 42, the token-mean absolute learner-to-behavior log ratio (log scale), the normalized ESS of the diagnostic sequence-level learner-to-behavior importance weights over each logged block of 2,048 responses (log scale), and the actor gradient norm (Appendix C.1 ). The mismatch diagnostics use mixed-reward-filtered training responses.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: MATH-500 learning curves by base-model difficulty on Qwen3-1.7B-Base. Left: mean@32 on the 410 problems the base model solves part of the time (

0<\hat{p}_{0}\leq 0.75

). Right: fraction of the 63 initially unsolved problems (no correct response among the base model’s 32 samples,

\hat{p}_{0}=0

) with at least one correct response among 32 samples. Both panels plot GRPO, AGRO, and GGPO in the default setting with seed 42 against optimizer updates.

Figure 6: Analytic coefficient sum for correct responses in a mixed-reward group. For

m=8

, curves show the signed sum

W_{\mathrm{correct}}=\sum_{i:r_{i}=1}c_{i}

, normalized to its value at

k=7

for each method. GRPO uses standardized reward advantages,

W_{\mathrm{correct}}\propto\sqrt{k(m-k)}

; AGRO uses leave-one-out advantages with equal learner-to-reference log ratios within the group,

W_{\mathrm{correct}}\propto k(m-k)/(m-1)

; GGPO uses Eq. ( 29 ) in the binary-reward limit

\beta\to 0

at fixed finite log ratios,

W_{\mathrm{correct}}=(m-k)/\sqrt{m-1}

Method	$K=1$	$K=4$	$K=8$	$K=16$	$K=32$
Base model	19.24	38.57	47.43	55.43	62.68
GRPO	42.55	56.10	61.38	65.74	69.39
AGRO	45.06	57.41	61.96	65.85	69.19
GGPO	46.60	60.91	66.22	70.46	73.90

Appendix

Table 3: Pass@

K

across five benchmarks on Qwen3-1.7B-Base. Problem-weighted pass@

K

(%) at the final 8,000-update checkpoint for training seed 42. The base model is evaluated under the same protocol. Bold marks the best method at each

K

Method ( $N$ )	MATH-500	AIME 2024	AIME 2025	OlympiadBench	Minerva Math	Avg.
GRPO ( $N=8$ )	$68.86\pm 0.32$	$9.03\pm 0.90$	$5.14\pm 0.63$	$32.13\pm 0.27$	$28.33\pm 0.40$	$42.64\pm 0.18$
GRPO ( $N=16$ )	68.56	8.96	5.00	31.87	28.15	42.39
GRPO ( $N=32$ )	68.43	8.96	5.00	31.76	28.07	42.28
GRPO ( $N=64$ )	68.28	8.85	5.00	31.62	27.98	42.15
GRPO ( $N=128$ )	$64.53\pm 0.85$	$7.71\pm 2.35$	$3.96\pm 1.51$	$28.30\pm 0.69$	$26.02\pm 1.01$	$39.02\pm 0.45$
GRPO+TIS ( $N=128$ )	$66.46\pm 0.62$	$8.30\pm 1.20$	$4.51\pm 1.05$	$30.12\pm 0.47$	$27.09\pm 0.64$	$40.69\pm 0.38$

Appendix

Table 4: Full evaluation across controlled policy staleness on Qwen3-1.7B-Base. Final-checkpoint mean@32 accuracy after 8,000 optimizer updates on five benchmarks (%), by synchronization delay

N

. Results at

N=8,128

are three-seed means;

N=8

reuses Table 1 , and

N=16,32,64

use seed 42. The final column gives the problem-weighted average. Sample standard deviations across seeds are reported for all columns at

N=8,128

Figure 7: Training-time diagnostics across the staleness ladder on Qwen3-1.7B-Base. Each panel plots one metric against synchronization delay

N

(equally spaced positions), averaged over the last 800 optimizer updates of each run: the token-mean K3 diagnostic of behavior–learner mismatch (

\times 10^{-3}

), the normalized ESS of the diagnostic sequence-level learner-to-behavior importance weights (Appendix C.1 ), policy entropy, and mean response length in tokens (GRPO orange squares, AGRO green triangles, GGPO blue circles). All diagnostics use seed 42; K3 and ESS are computed on mixed-reward-filtered training responses. K3 and ESS use logarithmic vertical axes. Corresponding accuracies are in Table 4 .

Optimizer updates	GRPO	GRPO+TIS	AGRO	GGPO
0	32.96	32.96	32.96	32.96
80	37.63 $\pm$ 0.38	54.58 $\pm$ 0.11	54.58 $\pm$ 0.35	54.83 $\pm$ 0.62
160	38.36 $\pm$ 0.27	57.93 $\pm$ 0.33	53.72 $\pm$ 0.44	59.31 $\pm$ 0.35
240	38.93 $\pm$ 0.31	58.28 $\pm$ 0.42	57.91 $\pm$ 0.73	59.79 $\pm$ 0.69
320	39.79 $\pm$ 0.32	54.51 $\pm$ 0.49	53.03 $\pm$ 0.68	55.83 $\pm$ 0.34
400	39.64 $\pm$ 0.72	42.71 $\pm$ 1.23	46.43 $\pm$ 0.71	48.31 $\pm$ 0.36

Appendix

Table 5: Offline MATH-500 accuracy across three seeds on Qwen3-1.7B-Base. Mean@32 accuracy (%) at each evaluated optimizer update, reported as the mean

\pm

sample standard deviation across training seeds 42, 1234, and 2027. Update 0 is the base model.

Method, $\beta$ , reference	MATH-500	AIME 2024	AIME 2025	OlympiadBench	Minerva Math	Avg.
GRPO, $\beta=0.01$ , fixed	68.86	9.03	5.14	32.13	28.33	42.64
GRPO, $\beta=0.01$ , $T_{\mathrm{ref}}=160$	68.45	9.27	4.90	31.79	28.57	42.40
GRPO, $\beta=0.01$ , $T_{\mathrm{ref}}=320$	68.36	9.06	4.69	32.29	27.99	42.48
AGRO, $\beta=0.01$ , fixed	66.61	8.13	4.06	31.64	27.48	41.48
AGRO, $\beta=0.01$ , $T_{\mathrm{ref}}=160$	70.68	9.27	6.56	35.73	29.30	45.07
AGRO, $\beta=0.01$ , $T_{\mathrm{ref}}=320$	69.94	8.13	4.58	34.07	28.47	43.86

Appendix

Table 6: Reference-refresh cadence and regularization on Qwen3-1.7B-Base. Final-checkpoint mean@32 accuracy after 8,000 optimizer updates on five benchmarks (%). “Fixed” keeps the initial reference throughout training;

T_{\mathrm{ref}}

is the refresh cadence in optimizer updates. The

\beta=0.01

rows at each method’s default cadence reuse the three-seed means from Table 1 ; the remaining conditions use seed 42. The final column uses the problem-weighted average of Table 1 . The compact main-text summary in Table 2 retains this seed convention.

Method	Metric	Fixed	$T_{\mathrm{ref}}=160$	$T_{\mathrm{ref}}=320$
AGRO	Log ratio (nats)	19.2	3.8	6.6
	Entropy	0.19	0.07	0.10
GGPO	Log ratio (nats)	6.0	0.6	0.7
	Entropy	0.35	0.19	0.18

Appendix

Table 7: Learner-to-reference log-ratio and entropy diagnostics on Qwen3-1.7B-Base. Mean sequence log ratio

\log[\pi_{\theta}/\pi_{\mathrm{ref}}]

over the responses in the training batch, and policy entropy, each averaged over the last 800 optimizer updates of the seed-42 run.

Figure 8: GGPO training under different reference-refresh cadences and regularization strengths on Qwen3-1.7B-Base. MATH-500 mean@32 accuracy against optimizer updates for

\beta\in\{0.01,0.1\}

with a fixed reference or refresh every 160 updates, and for

\beta=0.01

with refresh every 320 updates. All curves use seed 42. Final results across all five benchmarks are reported in Table 6 .

Figure 9: Within-success weight concentration for GGPO on Qwen3-1.7B-Base. Normalized ESS of the weights restricted to successful responses, for groups with at least two successes, at

\beta=0.1

with seed 42. Blue uses a fixed reference; orange refreshes every 160 optimizer updates. Higher values indicate more uniform weights among successful responses.

Group size	$K=1$	$K=4$	$K=8$	$K=16$	$K=32$
$m=4$	46.51	60.73	65.97	70.12	73.17
$m=8$	46.60	60.91	66.22	70.46	73.90
$m=16$	46.88	61.43	66.80	71.25	75.10

Appendix

Table 8: Evaluation across GGPO group sizes on Qwen3-1.7B-Base. Problem-weighted best@

K

(%) over the five benchmarks at the final 8,000-update checkpoint for training seed 42. For binary rewards, best@

K

equals pass@

K

. The

m=8

row reuses the GGPO results in Table 3 . All runs use 32 groups per learner minibatch.

Figure 10: Token clipping improves early training progress. Mean training reward (higher is better) on Qwen3-1.7B-Base against optimizer updates, with

N=8

m=8

\beta=0.01

T_{\mathrm{ref}}=160

, and seed 42. Blue shows the original GGPO update; orange shows the detached-coefficient variant with token log-ratio clipping at

\pm 0.05

. Rewards are measured on mixed-reward-filtered training data. Faint lines show values aggregated over eight updates; bold lines show trailing 160-update means. Clipping raises early training reward, with a smaller difference later in training.

Reinforcement Learning with Verifiable Rewards

Reinforcement learning with verifiable rewards trains models using outcomes checked by code, proofs, simulators, or structured criteria.

See all 685 papers

Oct 9, 2025

Group Contrastive Policy Optimization: Learning Beyond the Self-Exploration Limit with Golden Answers

Reinforcement learning with verifiable rewards (RLVR) has become a powerful paradigm for improving the reasoning abilities of large language models, yet group-based methods lose their learning signal when every sampled response receives zero reward. We propose Group Contrastive Policy Optimization (GCPO), a hybrid objective that preserves standard on-policy optimization for informative rollout groups and activates a proximal contrastive objective only when exploration completely fails. The auxiliary objective pairs an offline golden reasoning trajectory with the current policy's failed responses, thereby providing a reliable positive reference and prompt-specific hard negatives without treating the golden trajectory as an on-policy rollout. Its old-policy token probabilities are evaluated through teacher forcing and used solely to constrain the update, rather than as an importance-sampling correction for the teacher distribution. A first-order analysis shows that the resulting update increases a length-normalized likelihood margin between reliable reasoning and observed failures. Experiments across two model scales and 10 core mathematical reasoning benchmarks demonstrate that GCPO improves training efficiency and final reasoning performance over competitive supervised and RLVR baselines.

Hao Wu, Wei Liu, Shizhe Xiang +3

RL for Language Model Reasoning LLM Reasoning Reinforcement Learning with Verifiable Rewards Contrastive RL Group Relative Policy Optimization

Ask PDF

May 13, 2026

Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective

Group Relative Policy Optimization (GRPO) is one of the most widely adopted RLVR algorithms for post-training large language models on reasoning tasks. We first show that GRPO admits an equivalent discriminative reformulation, in which policy optimization maximizes the expected score gap between verified positive and negative rollouts. This reformulation reveals two objective-level limitations: likelihood-misaligned surrogate scores, in which clipped ratio-based scores are optimized rather than the sequence likelihoods that govern generation, and score-insensitive credit assignment, in which rollout-level credit does not reflect the current score gaps between positive and negative rollouts. To address these limitations, we propose ConSPO, a Contrastive Sequence-level Policy Optimization method that uses length-normalized sequence log-probabilities as rollout scores and contrasts verified positive rollouts against negative distractors within the same group. ConSPO optimizes a group-wise InfoNCE-style objective to adaptively strengthen updates for poorly separated positives and high-scoring negatives, together with a curriculum-scheduled margin that preserves separation pressure as training progresses. Experiments across diverse settings show that ConSPO outperforms strong baselines on challenging reasoning benchmarks. Code will be released upon paper acceptance.

Feng Zhang, Xinhong Ma, Ziqiang Dong +5

Beijing Institute of Technology · Qwen Business Unit of Alibaba · The Chinese University of Hong Kong, Shenzhen +1

RL for Language Model Reasoning Group Relative Policy Optimization Policy Optimization Reinforcement Learning with Verifiable Rewards RL Post-Training Contrastive RL

Ask PDF

May 17, 2026

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning

Group Relative Policy Optimization (GRPO) has been a key driver of recent progress in reinforcement learning with verifiable rewards (RLVR) for large language models, but it is typically trained in a low-staleness, near-on-policy regime that incurs substantial system overhead. We ask a simple question: How off-policy can GRPO be? We show that GRPO-style algorithms can tolerate substantially larger rollout staleness than previously assumed, and propose Mu-GRPO, an RL training framework that organizes training into a small number (e.g., four) of large sequential generation-optimization stages. This design induces high rollout staleness while greatly reducing rollout-optimization switching overhead. To stabilize learning under stale data, Mu-GRPO combines relaxed clipping, which preserves useful stale-rollout gradients, with negative-advantage veto, which removes destabilizing post-trigger suffix updates in negative-advantage responses. Across five language models and multiple math reasoning benchmarks, Mu-GRPO matches or exceeds the performance of standard GRPO while achieving around 2x speedup in wall-clock training time, establishing a substantially improved performance-efficiency trade-off for LLM reinforcement learning.

Minghao Tian, Yunfei Xie, Chen Wei

Rice University

RL for Language Model Reasoning Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards RL Post-Training Policy Learning Off-Policy RL

Ask PDF

Abstract

Reinforcement Learning with Verifiable Rewards

Explore similar work

Group Contrastive Policy Optimization: Learning Beyond the Self-Exploration Limit with Golden Answers

Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning