Is it RL or Data? A Controlled Comparison of GRPO and Rejection-Sampling Fine-Tuning

Lakshya Chaudhry

cs.LGOct 19, 2026

Is it RL or Data? A Controlled Comparison of GRPO and Rejection-Sampling Fine-Tuning

Authors: Lakshya Chaudhry

Organizations: Purdue University

Abstract

Bauer et al. (2026) showed that in low-data reinforcement learning with verifiable rewards, training GRPO on mixed-difficulty prompts gives up to 5x the sample efficiency of easy-only prompts. They did not test whether this benefit is specific to GRPO, or whether another post-training method given the same data would obtain it too. We test this on their procedural counting task with Qwen3-4B-Base and LoRA, comparing GRPO to rejection-sampling fine-tuning (RFT) on the same prompts, verifier and token cap. Six primary arms cross the two methods with three prompt sets (Easy, Mixed, and a Curated subset of prompts the base model sometimes solves, which makes GRPO's implicit filter explicit). We use three seeds per arm. Hypotheses and falsifiers were registered before training. To rule out a poorly tuned baseline, RFT receives a 27-run hyperparameter sweep, while GRPO uses Bauer et al.'s recipe unchanged. GRPO outperforms RFT at every prompt set: by 10.9 points on held-out test problems with curated prompts, and, in post hoc comparisons with matched rollout budgets, by 8.4 and 9.2 points on the easy and mixed prompts. On test the advantage survives the least favorable treatment of truncated completions; out of distribution, truncation prevents a conclusion. Curating prompts does not help RFT (-2.7 points). Mixed-difficulty data helps RFT (+3.7 points), while for GRPO the evidence is inconclusive. A secondary iterated-RFT arm, which resamples from the current policy, recovers 3.2 of the 10.9-point gap (about 30%). These are controlled measurements on one synthetic task family and one base model with three seeds per arm.

Figures & tables

Arm	Seeds	Prompts	Rollouts (avail. / used)	Train tokens	Opt. steps
RFT-Easy	3	100	19,200 / 13,834	47.2M	6,920
RFT-Mixed	3	100	19,200 / 7,961	15.9M	1,992
RFT-Curated	3	73	14,016 / 6,653	13.4M	1,664
GRPO-Easy	3	100	19,200 / 19,200	12.4M	300
GRPO-Mixed	3	100	19,200 / 19,200	23.0M	300
GRPO-Curated	3	73	19,200 / 19,200	20.6M	300

Table 1: Arms and budgets (means per seed). Rollouts: sampled completions available / used for training. RFT-Mixed has a correct sample for only 95 of its 100 prompts.

	test_300		ood_hard_200
Arm	acc.	trunc.	acc.	trunc.
Base	67.3	4.3	25.0	13.0
RFT-Easy	65.9 $\pm$ 1.1	2.4	25.5 $\pm$ 0.5	10.5
RFT-Mixed	69.6 $\pm$ 1.0	2.0	27.3 $\pm$ 1.2	12.2
RFT-Curated	66.9 $\pm$ 1.1	3.1	28.5 $\pm$ 1.8	14.8
GRPO-Easy	74.3 $\pm$ 3.9	3.0	33.8 $\pm$ 3.1	26.2

Table 2: Greedy accuracy (%), mean

\pm

sd over 3 seeds, with the share of completions cut at the 4,352-token cap (trunc.). ‡ All three seeds exceed 5% truncation on test_300 . Base, controls and the reference model are single runs.

Comparison (A $-$ B)	$\Delta$ test_300	$2\sigma$	$\Delta$ ood_hard_200	Crit.
Training signal: GRPO $-$ RFT on the same prompts
GRPO-Curated $-$ RFT-Curated	$+10.9$ [ $+6.7$ , $+15.1$ ]	2.8	$+3.0$ [ $-2.5$ , $+8.5$ ]	✓
GRPO-Easy $-$ RFT-Easy	$+8.4$ [ $+4.8$ , $+12.1$ ]	5.8	$+8.3$ [ $+3.7$ , $+13.0$ ]	✓
GRPO-Mixed $-$ RFT-Mixed	$+9.2$ [ $+4.8$ , $+13.7$ ]	1.6	$+3.5$ [ $-2.0$ , $+9.0$ ]	✓
Prompt curation
RFT-Curated $-$ RFT-Mixed	$-2.7$ [ $-5.2$ , $0.0$ ]	2.1	$+1.2$ [ $-2.5$ , $+5.3$ ]	–

Table 3: Paired comparisons (A

-

B, pp), mean over seeds with a paired bootstrap 95% CI.

2\sigma

: twice the pooled seed sd. ✓: meets the success criterion (

|\Delta_{\text{test}}|>2\sigma

and the same sign on OOD). The easy and mixed GRPO

-

RFT rows and the iterated-RFT rows are post hoc.

Figure 1: Greedy accuracy on test_300 by difficulty tier (100 problems per tier). Dots are seeds with 95% bootstrap CIs; black bars are the mean

\pm

sd over seeds. Controls and Base are single runs. Each row has its own y-axis range. RFT and GRPO are close on easy problems and separate on hard ones.

Figure 2: Share of GRPO groups whose eight completions all received the same reward, and so contribute no gradient, over training. One line per seed, 25-step trailing mean. Columns split the prompts by difficulty tier; the easy prompt set contains only easy prompts.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Split	$n$	easy / medium / hard	steps
train_easy_100	100	100 / 0 / 0	2–5
train_mixed_100	100	33 / 33 / 34	2–5
train_curated	73	26 / 33 / 14	2–5
val_mixed_100	100	33 / 33 / 34	2–5
test_300	300	100 / 100 / 100	2–5
ood_hard_200	200	14 / 47 / 139	6–8

Appendix

Table 4: Split sizes and tier composition. ood_hard_200 is tiered after generation but not filtered.

Cell	$n$ correct	median	$p_{99}$	over 2,048 (%)
S2	139	261	1,183	0.0
S3	172	346	1,690	1.2
S4	152	463	1,431	0.0
S5	133	552	2,114	1.5
M2	95	400	2,675	4.2
M3	103	392	3,478	2.9

Appendix

Table 5: Per-cell length statistics of correct base-model completions (tokens) used to set the cap. Cell = range band and total steps.

Shared
Model	Qwen/Qwen3-4B-Base , bf16
LoRA	rank 64, $\alpha{=}16$ , dropout 0; q, k, v, o, gate, up, down
Optimizer	AdamW, weight decay 0, gradient clip 1.0
Schedule	cosine, 10% warmup
Sampling	$T{=}1.0$ , top- $p$ 1.0, cap 4,352 completion / 4,096 prompt tokens
Seeds	1, 2, 3 (LoRA init, data order, sampling)

Appendix

Table 6: Training settings. Values marked ∗ are library defaults recorded from the resolved configuration.

	lr $10^{-5}$			lr $5{\times}10^{-5}$			lr $10^{-4}$
Epochs	2	4	8	2	4	8	2	4	8
RFT-Easy	59	64	61	63	62	67	61	64	63
RFT-Mixed	62	62	66	65	67	63	62	67	57
RFT-Curated	62	66	65	63	64	60	64	64	58

Appendix

Table 7: RFT sweep: val_mixed_100 greedy accuracy (%,

n=100

, seed 1) for each learning rate and epoch count. Bold: selected (ties broken by fewer epochs, then lower learning rate).

	test_300			ood_hard_200
Arm	seed 1	seed 2	seed 3	seed 1	seed 2	seed 3
RFT-Easy	66.3 (2.3)	64.7 (3.3)	66.7 (1.7)	25.0 (14.0)	26.0 (9.5)	25.5 (8.0)
RFT-Mixed	70.7 (2.7)	68.7 (1.7)	69.3 (1.7)	28.0 (12.0)	28.0 (11.0)	26.0 (13.5)
RFT-Curated	67.3 (2.7)	65.7 (3.3)	67.7 (3.3)	30.0 (14.0)	29.0 (15.5)	26.5 (15.0)
GRPO-Easy	77.7 (4.3)	70.0 (3.3)	75.3 (1.3)	34.5 (38.0)	30.5 (21.0)	36.5 (19.5)
GRPO-Mixed	78.7 (6.7)	79.3 (10.3)	78.3 (9.0)	29.5 (46.5)	29.5 (54.5)	33.5 (41.5)

Appendix

Table 8: Greedy accuracy (%) by seed, with the share of completions truncated at the cap in parentheses. Mean

\pm

sd is in Table 2 .

Figure 3: Mean completion length (top) and share of completions truncated at the 4,352-token cap (bottom) under greedy decoding. Each dot is one seed (three per arm; one for Base, C1 and C2). Colors: gray Base, blue RFT, orange GRPO, purple iterated RFT, green controls. GRPO-Mixed and GRPO-Curated write longer completions and are truncated more often than RFT, especially on the out-of-distribution set.

Comparison (A $-$ B)	observed	lower	upper
GRPO-Curated $-$ RFT-Curated	$+10.9$	$+7.8$	$+18.4$
GRPO-Easy $-$ RFT-Easy	$+8.4$	$+6.0$	$+11.4$
GRPO-Mixed $-$ RFT-Mixed	$+9.2$	$+7.2$	$+17.9$
RFT-Curated $-$ RFT-Mixed	$-2.7$	$-4.7$	$+0.4$
RFT-Mixed $-$ RFT-Easy	$+3.7$	$+1.2$	$+5.7$
GRPO-Mixed $-$ GRPO-Easy	$+4.4$	$+1.4$	$+13.1$

Appendix

Table 9: Truncation bounds on test_300 (pp): the observed difference and its range when truncated completions are counted against A (lower) or against B (upper). On ood_hard_200 every interval includes zero.

Figure 4: pass@

k

on the first 100 test_300 problems, from 64 samples per problem at

T{=}1.0

(unbiased estimator). Thin lines are seeds; thick lines are the mean over seeds; shaded bands are 95% bootstrap CIs over the 100 problems, for Base (gray) as well. Arms are colored: RFT blue, GRPO orange, iterated RFT purple, controls green. C1 and C2 are single seeds (one curve, no seed lines). GRPO lifts pass@

k

at small

k

and meets Base, within the bands, at

k=64

Arm	test mean@8	OOD mean@8	pass@8	pass@64
Base	42.5	17.3	80.3	98.0
RFT-Easy	53.5	24.3	83.3	94.0
RFT-Mixed	54.4	23.5	84.9	96.3
RFT-Curated	51.2	21.6	83.0	95.3
GRPO-Easy	70.1	33.0	87.4	95.7
GRPO-Mixed	77.1	31.1	91.5	96.3

Appendix

Table 10: Sampled accuracy (%): mean@8 at

T{=}1.0

on test_300 and ood_hard_200 , and pass@8 / pass@64 on the first 100 test problems. Mean over seeds; controls and Base are single runs.

Arm	training (selected)	sweep	evaluation	total
RFT-Easy	21.3	30.1	1.8	53.1
RFT-Mixed	7.0	22.1	1.9	31.0
RFT-Curated	5.9	18.5	1.9	26.3
GRPO-Easy	15.5	–	3.0	18.5
GRPO-Mixed	27.1	–	5.0	32.0
GRPO-Curated	25.1	–	4.4	29.4

Appendix

Table 11: Metered GPU-hours (H100) per arm. “Sweep” is the eight non-selected RFT configurations.

Fine-Tuning

Fine-tuning adapts pretrained models to new tasks, domains, and behaviors by updating some or all of their parameters.

See all 250 papers

Mar 22, 2026

Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts

Reinforcement learning with verifiable rewards (RLVR) plays a crucial role in expanding the capacities of LLM reasoning, but GRPO-style training is dominated by expensive rollouts and wastes compute on unusable prompts. We propose Prompt Replay, an overhead-free online data selection method for GRPO that reuses prompts only (not trajectories), to preserve on-policy optimization. After each step, we insert prompts with medium difficulty into a buffer, and prioritize prompts closer to a pass rate of 0.5 (half answers correct, half wrong) to maximize the advantage, thus learning signal. Training batches are formed by mixing reused prompts with fresh samples, with cooldown steps and max reuse times controlling aggressiveness vs risk of overfitting. Across multiple model families (Llama-3.2- 3B, Qwen3-8B) and training datasets (Dolci, Polaris), evaluated using average accuracy on six standard math benchmarks, Prompt Replay reduces zero-variance prompts, increases mean absolute advantage and shows faster initial accuracy gains. Yet, it plateaus and converges with the baseline, as too aggressive configuration was used. The method is most efficient when the rollouts are the primary bottleneck and the dataset is difficult for the model. We additionally observe that Qwen2.5-Math can exhibit spurious-reward effects that invalidates ablations, raising a warning signal for using it as a sole testbed for GRPO method research.

Andrei Baroian, Rutger Berger

Leiden Institute of Advanced Computer Science Leiden University

Prompt Selection for RL RL Post-Training Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards

Ask PDF

May 17, 2026

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning

Group Relative Policy Optimization (GRPO) has been a key driver of recent progress in reinforcement learning with verifiable rewards (RLVR) for large language models, but it is typically trained in a low-staleness, near-on-policy regime that incurs substantial system overhead. We ask a simple question: How off-policy can GRPO be? We show that GRPO-style algorithms can tolerate substantially larger rollout staleness than previously assumed, and propose Mu-GRPO, an RL training framework that organizes training into a small number (e.g., four) of large sequential generation-optimization stages. This design induces high rollout staleness while greatly reducing rollout-optimization switching overhead. To stabilize learning under stale data, Mu-GRPO combines relaxed clipping, which preserves useful stale-rollout gradients, with negative-advantage veto, which removes destabilizing post-trigger suffix updates in negative-advantage responses. Across five language models and multiple math reasoning benchmarks, Mu-GRPO matches or exceeds the performance of standard GRPO while achieving around 2x speedup in wall-clock training time, establishing a substantially improved performance-efficiency trade-off for LLM reinforcement learning.

Minghao Tian, Yunfei Xie, Chen Wei

Rice University

RL for Language Model Reasoning Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards RL Post-Training Policy Learning Off-Policy RL

Ask PDF

Sep 5, 2026

DataFlex-RL: An Evaluation Platform for RLVR Data Policies

Data policies for reinforcement learning with verifiable rewards (RLVR) determine which rollouts are used, how strongly they are weighted, and which domains contribute to subsequent training batches. We introduce DataFlex-RL, an evaluation platform for comparing these choices under a common GRPO recipe. Our primary experiment evaluates 13 configurations across 12 matched seeds using Qwen2.5-7B-Base and 12 mathematics, logic, and science benchmarks. Uniform GRPO improves the domain-balanced average accuracy by 7.76 percentage points over the untrained checkpoint. None of the eight rollout-selection or reweighting methods achieves a paired 95% confidence interval that excludes zero relative to uniform sampling, and none of the three adaptive mixtures outperforms a fixed equal mixture at the same level of precision. A corrected 12-seed extension on Llama-3.1-8B-Base places the additional methods on the same score scale as the original controls, but does not reveal a consistent winner in terms of observed mean performance. We also quantify evaluation sensitivity by rescoring nine Qwen2.5-7B-Instruct runs using a math-heavy six-benchmark summary, consisting of five mathematics benchmarks and GPQA-Diamond but no logic benchmark, and comparing it with the domain-balanced 12-benchmark summary. The resulting rankings are negatively correlated, with a correlation coefficient of -0.33, whereas summaries that retain all 12 benchmarks largely agree. Across the controlled settings studied here, changing the data policy measurably changes the training process but does not produce a reproducible improvement over uniform training.

Hao Liang, Mingrui Chen, Hengyi Feng +2

Peking University · Zhongguancun Academy · UCAS +1

Reinforcement Learning RL Post-Training Reinforcement Learning with Verifiable Rewards

Ask PDF

Abstract

Fine-Tuning

Explore similar work

Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning

DataFlex-RL: An Evaluation Platform for RLVR Data Policies