Prompt Dominance and Asymmetric Verifier Costs: Empirical Ablations of GRPO at 1B Scale on GSM8K

Yi Hou

cs.LGOct 4, 2026

Prompt Dominance and Asymmetric Verifier Costs: Empirical Ablations of GRPO at 1B Scale on GSM8K

Authors: Yi Hou

Organizations: University of Chinese Academy of Sciences Beijing, China

Abstract

This paper studies GRPO at 1B scale from both directions: what estimator choices do to the learning signal, and what a degraded reward signal does to what is learned. We train OLMo-2-0425-1B on GSM8K with a from-scratch implementation and measure both sides in controlled sweeps, including a verifier-quality experiment that degrades the training reward and the test-time selector identically. Four results stand out. The prompt is the first-order decision: the zero-shot prompt leaves the base model at 0.08% (its outputs are degenerate continuations, not wrong answers), so almost no group carries a gradient, and training succeeds because the 3-shot prompt reaches 18.3%. At this scale the estimator variants sit within seed noise, with Dr. GRPO ahead on both seeds. In the off-policy regime, clipping is the whole story: training on data without a clipped ratio loses 4-6 points relative to the on-policy reference, while GRPO-style clipping and GSPO recover the loss entirely. Finally, the same weak verifier is far cheaper in RL than in test-time selection: a 10%-flip verifier leaves RL's attainable gain intact (91% and 106% retained across two seeds) where selection retains 57%, and a format-only verifier leaves RL with 16-30% of its gain and selection with essentially nothing. Flip noise acts as an affine transform on the expected reward, and the group-normalized advantage with Adam's rescaling removes it exactly; the residual is a second-order variance effect that the matched-step comparison at a 30% flip rate tests.

Figures & tables

Prompt	Accuracy	Format	Formatted, wrong	Length
question_only	0.23%	17.4%	17.2%	273
r1_zero (0-shot)	0.08%	62.6%	62.5%	97
r1_zero (3-shot)	18.27%	94.2%	75.9%	108

Table 1: Base-model GSM8K test accuracy by prompt (1,319 questions, sampled at temperature 1.0). “Format” is the share of responses the grader accepts as well-formed; “length” is the mean response length in tokens.

Figure 1: The standard run (200 steps, 3-shot prompt). Solid lines: greedy decoding; dashed: sampling at temperature 1.0. Both seeds are shown. The dotted line is the 3-shot prompting baseline from Table 1 .

Figure 2: Learning-rate sweep (80 steps, seed 0, greedy decoding).

3\times 10^{-5}

peaks at step 30 and then falls by 10 points: divergence, not slow convergence;

3\times 10^{-6}

has not converged by step 80.

Figure 3: question_only under GRPO (80 steps, both seeds, greedy decoding). The same configuration that more than doubles accuracy with a 3-shot prompt barely moves here, and the two seeds disagree by a factor of seven.

Figure 4: Four one-change variants of the standard estimator, 80 steps, both seeds, final greedy accuracy. Each point is one seed; the bar is the two-seed mean. The dashed line is the standard run’s own 80-step value (0.526, seed 0). Variant definitions in the text.

Figure 5: Off-policy corrections (80 steps, both seeds, final greedy accuracy). Each run consumes a rollout batch as 32 sequential training steps, so the last step trains on data 32 steps stale. Only the correction scheme differs.

Figure 6: Test-time selection over the fixed sample population (512 questions, 32 samples each). Best-of-

n

re-ranks the same samples under verifiers of decreasing quality; “oracle” is the coverage ceiling (pass@

n

, the unbiased estimator of [ 2 ] ). The horizontal line is the no-selection accuracy (n=1).

Figure 7: Training with a weak verifier (80 steps, both seeds), evaluated with the true grader on the 1,319-question test split. Each point is one seed, the bar is the mean; the dotted line is the untrained base model under the same protocol. The

30\%

-flip arm is absent here: it is compared on the internal validation split (Section 6.2 ).

Figure 8: Retained fraction of the attainable gain as the verifier degrades, for the two uses of the same verifier. Selection: best-of-32 minus the base rate, over the coverage ceiling minus the base rate, on the 512-question population of Figure 6 . RL: the trained arm minus the same-protocol prompting baseline, over the exact arm minus that baseline, on the test split; each dot is one seed. The RL series stops at

\epsilon=0.1

because the 30%-flip arm has no test-set evaluation; it is compared on the internal validation split in Section 6.2 . The dashed line is the closed form for the expected-gradient scaling of RL under flip noise,

(1-2\epsilon)

Explore similar work

VeriGate: Verifier-Gated Step-Level Supervision for GRPO
May 28, 2026Aakriti Agrawal, Minghui Liu, Furong HuangProcess Reward Models RL for Language Model Reasoning
University of Maryland, College Park
Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts
Mar 22, 2026Andrei Baroian, Rutger BergerRL Post-Training Group Relative Policy Optimization
Leiden Institute of Advanced Computer Science Leiden University
F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
Feb 6, 2026Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov +4Reinforcement Learning RL for Language Model Reasoning
T-Tech · Saint Petersburg Electrotechnical University “LETI”

May 28, 2026J/K move · Enter open · S save

VeriGate: Verifier-Gated Step-Level Supervision for GRPO

Aakriti Agrawal, Minghui Liu, Furong Huang

University of Maryland, College Park

Process Reward Models RL for Language Model Reasoning Group Relative Policy Optimization Credit Assignment in RL Reinforcement Learning with Verifiable Rewards Step-Level Credit Assignment

Open paper AskPDF arXiv

Group Relative Policy Optimization (GRPO) is an effective recipe for training reasoning models with verifier-based outcome rewards, but its supervision is sparse: when all sampled trajectories for a prompt receive the same verifier reward, the group-relative advantage collapses to zero and learning stalls. Outcome-only rewards also provide no step-level credit assignment, limiting exploration and making it harder to learn robust reasoning. We present VeriGate (Verifier-Gated Step-Level GRPO), a verifier-gated extension of GRPO that addresses these limitations with three design choices. First, VeriGate keeps the verifier in charge whenever verifier rewards induce a meaningful preference among sampled trajectories, and uses process supervision only when verifier rewards are degenerate. Second, instead of collapsing Process Reward Model (PRM) step scores into a single trajectory reward, VeriGate converts them into future-cumulated rewards to assign continuation-aware credit. Third, VeriGate transforms these rewards into group-normalized token-level advantages, restoring informative gradients and fine-grained credit assignment while remaining less susceptible to reward hacking than methods that optimize aggregated PRM scores. Empirically, training on MATH with 1.5B and 7B Qwen2.5-Instruct models and evaluating on six reasoning benchmarks, VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively, substantially reduces zero-gradient failures, decreases reward-hacking behavior, and improves reasoning quality relative to outcome-only GRPO and PRM-as-outcome baselines.

Abstract

Explore similar work

VeriGate: Verifier-Gated Step-Level Supervision for GRPO

Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare