Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO

Jiahua Yang; Zhiwei Yang; Xianpeng Zhang; Dongyu Chen; Xing Chen; Tianhuang Su; Haonan Lu; Quanlong Guan

cs.AISep 29, 2026

Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO

Authors: Jiahua Yang, Zhiwei Yang, Xianpeng Zhang, Dongyu Chen, Xing Chen, Tianhuang Su, Haonan Lu, Quanlong Guan, +2 more

Organizations: Guangdong Institute of Smart Education, Jinan University, Guangzhou, China · OPPO AI Center, Shenzhen, China · Ragentile Intelligence Inc, Edmonton, Canada

Abstract

Recently, Group Relative Policy Optimization (GRPO) and its variants have been developed for policy optimization and demonstrated notable performance gains. However, these methods usually incur substantial computational overhead due to per-question multi-rollout sampling and repeated per-token probability evaluation across rollouts. Furthermore, low-information or highly homogeneous trajectories can degrade downstream learning signal efficiency, hindering model optimization and limiting final performance. To address these issues, we propose FastRL, a novel plug-and-play reinforcement learning framework that simultaneously improves training efficiency and the effectiveness of policy learning. Specifically, 1) We introduce an advantage-aware pruning strategy to selectively preserve high-advantage trajectories while maximizing inter-trajectory gradient diversity. 2) Then, we design an adaptive rollout sampling mechanism to dynamically adjust the sampling scale across different training stages based on historical pruning distributions, balancing exploration adequacy and computational efficiency. Experiments demonstrate that FastRL can be seamlessly integrated into GRPO, DAPO, and GSPO variants, achieving an average 2.07 $\times$ training speedup on Geometry3K and GeoQA8K-R1V, along with an approximately 1.64% improvement in average accuracy on visual reasoning benchmarks. Source codes will be available at https://github.com/Nicozwy/FastRL.

Figures & tables

Figure 1: Comparison of training efficiency and out-of-domain performance across reinforcement learning methods using Qwen2.5-VL-7B-Instruct on Geometry3K (left) and GeoQA8K-R1V (right).

Figure 2: The proposed FastRL framework. (a) Maximizing Differential Advantage Pruning (MDAP) identifies and prunes uninformative trajectories (i.e., $\theta_{ij}<\tau$ ) and homogeneous trajectories (i.e., $A_{i}=A_{j}$ ); (b) Adaptive Rollout Sampling (ARS) dynamically adjusts the rollout budget for each iteration based on the accumulated pruning distribution.

Dataset	Method	In-domain	MathVerse	MathVision	MathVista	WeMath	Avg( $\uparrow$ )	Train-Time( $\downarrow$ )	Speed( $\uparrow$ )
Qwen2.5-VL-7B-Instruct
Geometry3K	GRPO [ 15 ]	53.74	43.38	26.25	66.90	68.74	51.32	29.72	1 $\times$
	+CPPO [ 6 ]	54.41	43.98	27.37	66.30	69.08	51.68	18.52	1.60 $\times$
	+GRESO [ 25 ]	53.24	42.08	26.09	64.90	67.93	50.25	27.29	1.09 $\times$
	+FastRL (Ours)	55.90	45.76	27.96	67.30	69.48	52.63	13.87	2.14 $\times$
	DAPO [ 21 ]	53.57	41.24	26.25	67.20	70.69	51.35	28.08	1 $\times$

Table 1: Performance comparison (%) on Geometry3K and GeoQA8K-R1V regarding accuracy. The bold numbers denote the best results. Avg denotes the average accuracy across out-of-domain datasets, and Speed denotes the training speed regarding GRPO.

\uparrow

indicates that the higher is better, while

\downarrow

indicates that the lower is better.

Dataset	Method	In-domain	MathVerse	MathVision	MathVista	WeMath	Avg( $\uparrow$ )	Train-Time( $\downarrow$ )	Speed( $\uparrow$ )
Qwen2.5-VL-7B-Instruct
Geometry3K	FastRL	55.90	45.76	27.96	67.30	69.48	52.63	13.87	2.14 $\times$
	w/o MDAP	53.41 ${}_{{\color[rgb]{1,0,0}\downarrow\text{\!2.49}}}$	45.00	27.04	66.30	69.31	51.91 ${}_{{\color[rgb]{1,0,0}\downarrow\text{\!0.72}}}$	25.85	1.15 $\times$ ${}_{{\color[rgb]{1,0,0}\downarrow\text{\!0.99$ \times $}}}$
	w/o ARS	54.01 ${}_{{\color[rgb]{1,0,0}\downarrow\text{\!1.89}}}$	45.00	27.99	67.10	68.97	52.27 ${}_{{\color[rgb]{1,0,0}\downarrow\text{\!0.36}}}$	14.41	2.06 $\times$ ${}_{{\color[rgb]{1,0,0}\downarrow\text{\!0.08$ \times $}}}$
GeoQA8K-R1V	FastRL	70.15	45.91	27.86	70.60	70.00	53.59	25.65	2.13 $\times$
	w/o MDAP	68.70 ${}_{{\color[rgb]{1,0,0}\downarrow\text{\!1.45}}}$	44.95	27.43	67.60	69.54	52.38 ${}_{{\color[rgb]{1,0,0}\downarrow\text{\!1.21}}}$	49.75	1.10 $\times$ ${}_{{\color[rgb]{1,0,0}\downarrow\text{\!1.03$ \times $}}}$

Table 2: Ablation study (%) of FastRL. “w/o MDAP” denotes FastRL without maximizing differential advantage pruning (with ARS simulated to match its standard dynamics for a fair comparison), and “w/o ARS” denotes FastRL without adaptive rollout sampling.

Figure 3: Training efficiency and parameter sensitivity of FastRL on Geometry3K (top) and GeoQA8K-R1V (bottom). (a1)-(a2) show the the rollout sampling scale $G_{t}$ ; (b1)-(b2) present the number of trajectories used for policy updates; (c1)-(c2) show the sensitivity to the hyperparameter $\tau$ .

Dataset	Method	In-domain	AIME2023	AIME2024	AIME2025	AIME2026	Avg( $\uparrow$ )	Train-Time( $\downarrow$ )	Speed( $\uparrow$ )
Llama3.1-8B-Instruct
MATH	GRPO [ 15 ]	72.32	10.00	10.00	6.67	3.33	7.50	21.16	1 $\times$
	DAPO [ 21 ]	72.72	6.67	10.00	6.67	3.33	6.67	19.72	1.07 $\times$
	GSPO [ 24 ]	72.56	6.67	10.00	0.00	0.00	4.17	18.89	1.12 $\times$
	CPPO [ 6 ]	71.92	3.33	10.00	0.00	3.33	4.17	11.44	1.85 $\times$
	GRESO [ 25 ]	71.60	3.33	6.67	3.33	0.00	3.33	17.35	1.22 $\times$

Table 3: Generalization across model architectures and datasets.

Dataset	Method	MathVerse	MathVision	MathVista	WeMath	avg@16( $\uparrow$ )
Geometry3K	GRPO [ 15 ]	60.53	62.96	82.50	90.34	74.08
	DAPO [ 21 ]	58.98	62.57	81.50	90.57	73.41
	GSPO [ 24 ]	59.21	62.30	81.90	90.11	73.38
	GRESO [ 25 ]	59.44	62.47	82.00	90.23	73.54
	CPPO [ 6 ]	60.81	63.19	82.20	90.69	74.22
	FastRL (Ours)	63.60	67.86	83.60	93.85	77.23

Table 4: Pass@16 results (%) on out-of-domain benchmarks. Base model: Qwen2.5-VL-7B-Instruct.

Pruning Strategy	MathVerse	MathVision	MathVista	WeMath	Avg( $\uparrow$ )	Speed( $\uparrow$ )
Random-pruning 0.25	43.98	27.40	66.90	68.85	51.78	1.27 $\times$
Random-pruning 0.50	45.35	26.74	66.60	67.70	51.59	1.60 $\times$
Random-pruning 0.75	42.79	26.45	64.60	66.49	50.08	2.10 $\times$
Advantage-only (Ours)	44.54	27.57	67.10	69.14	52.09	2.35 $\times$
FastRL (Ours)	45.76	27.96	67.30	69.48	52.63	2.14 $\times$

Table 5: Ablation (%) of pruning strategies on Geometry3K using Qwen2.5-VL-7B-Instruct.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Multimodal					Unimodal
Dataset	Sum	Train	Test	Modality	Dataset	Sum	Train	Test	Modality
Geometry3K	2,702	2,101	601	Text + Image	MATH	12,500	11,250	1,250	Text
GeoQA8K-R1V	8,785	8,031	754	Text + Image	AIME2023	30	-	30	Text
MathVerse	3,940	-	3,940	Text + Image	AIME2024	30	-	30	Text
MathVision	3,040	-	3,040	Text + Image	AIME2025	30	-	30	Text
MathVista (mini)	1,000	-	1,000	Text + Image	AIME2026	30	-	30	Text

Appendix

Table 6: Statistics of the datasets used in our experiments.

Figure 4: Examples from the three training datasets: Geometry3K, GeoQA8k-R1V, and MATH.

Figure 5: Correlation analysis between token-level Jaccard similarity and gradient cosine similarity of trajectories with the same advantage for each question. The left plot corresponds to the Geometry3K dataset, while the right plot corresponds to the GeoQA8k-R1V dataset. The high Pearson and Spearman correlation coefficients demonstrate a strong positive correlation between the two metrics across both datasets.

Figure 6: Training dynamics and computational overhead analysis of Qwen2.5-VL-7B-Instruct on Geometry3K and GeoQA8K-R1V. We compare FastRL (ours) with GRPO, DAPO, CPPO, GSPO, and GRESO across three key metrics: total token consumption (left), policy entropy (middle), and accuracy (right).

Dataset	$\tau$	In-domain	MathVerse	MathVision	MathVista	WeMath	Avg( $\uparrow$ )	Train-Time( $\downarrow$ )	Speed( $\uparrow$ )
Qwen2.5-VL-7B-Instruct
Geometry3K	0(GRPO)	53.74	43.38	26.25	66.90	68.74	51.32	29.72	1 $\times$
	0.2	54.08	44.37	27.37	67.10	69.02	51.97	29.33	1.01 $\times$
	0.4	54.58	44.80	27.99	67.00	68.68	52.18	25.64	1.16 $\times$
	0.6	55.24	44.75	28.13	66.90	69.31	52.27	21.26	1.40 $\times$
	0.8	55.90	45.76	27.96	67.30	69.48	52.63	13.87	2.14 $\times$

Appendix

Table 7: Supplementary Parameter Sensitivity Analysis.

Dataset	Method	timing_step	timing_update_actor	timing_ref	timing_old	timing_gen
Geometry3K	GRPO [ 15 ]	534.96	319.37	70.14	55.44	87.05
Geometry3K	FastRL (Ours)	249.66	121.07	26.59	21.02	79.53
GeoQA8K-R1V	GRPO [ 15 ]	423.10	240.95	58.58	61.51	60.41
GeoQA8K-R1V	FastRL (Ours)	198.58	100.82	24.51	25.74	46.07

Appendix

Table 8: Average per-stage training time (seconds) of FastRL and GRPO on Qwen2.5-VL-7B-Instruct.

Method	General(en)		General(cn)		Efficiency
	center-in	mean-IoU	center-in	mean-IoU	Train-Time (h)( $\downarrow$ )	Speed( $\uparrow$ )
GRPO [ 15 ]	95.58	89.03	94.04	87.19	49.21	1.00 $\times$
FastRL (Ours)	97.33	89.19	94.91	88.45	28.01	1.76 $\times$

Appendix

Table 9: FastRL vs. GRPO on the grounding task under continuous rewards. The base model is Qwen3.5-9B.

Explore similar work

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
May 17, 2026Minghao Tian, Yunfei Xie, Chen WeiLarge Language Model Reinforcement Learning Flow-Grpo
Rice University
Group Adaptive Clipping Policy Optimization
Aug 31, 2026Sheng Jia, Xiao Wang, Shiva Prasad Kasiviswanathan +1Gradient Clipping Reinforcement Learning With Verifiable Reward
University of Toronto · Amazon
Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
May 20, 2026Xixiang He, Qiyao Sun, Ao Cheng +5Reinforcement Learning With Verifiable Reward Collapse
National University of Defense Technology, Changsha, Hunan, China · Intelligent Game and Decision Lab, Beijing, China · The Chinese University of Hong Kong, Shenzhen, Guangdong, China

May 17, 2026, cs.LGJ/K move · Enter open · S save

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning

Minghao Tian, Yunfei Xie, Chen Wei

Rice University

Large Language Model Reinforcement Learning Flow-Grpo Offline Reinforcement Learning Off-Policy Learning

Open paper AskPDF arXiv

Group Relative Policy Optimization (GRPO) has been a key driver of recent progress in reinforcement learning with verifiable rewards (RLVR) for large language models, but it is typically trained in a low-staleness, near-on-policy regime that incurs substantial system overhead. We ask a simple question: How off-policy can GRPO be? We show that GRPO-style algorithms can tolerate substantially larger rollout staleness than previously assumed, and propose Mu-GRPO, an RL training framework that organizes training into a small number (e.g., four) of large sequential generation-optimization stages. This design induces high rollout staleness while greatly reducing rollout-optimization switching overhead. To stabilize learning under stale data, Mu-GRPO combines relaxed clipping, which preserves useful stale-rollout gradients, with negative-advantage veto, which removes destabilizing post-trigger suffix updates in negative-advantage responses. Across five language models and multiple math reasoning benchmarks, Mu-GRPO matches or exceeds the performance of standard GRPO while achieving around 2x speedup in wall-clock training time, establishing a substantially improved performance-efficiency trade-off for LLM reinforcement learning.

Abstract

Explore similar work

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning

Group Adaptive Clipping Policy Optimization

Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation