Group Relative Policy Optimization (GRPO) is effective for training language models on complex reasoning. However, since the objective is defined relative to a group of sampled trajectories, extended deliberation can create more chances to realize relative gains, leading to inefficient reasoning and overthinking, and complicating the trade-off between correctness and rollout efficiency. Controlling this behavior is difficult in practice, considering (i) Length penalties are hard to calibrate because longer rollouts may reflect harder problems that require longer reasoning, penalizing tokens risks truncating useful reasoning along with redundant continuation; and (ii) supervision that directly indicates when to continue or stop is typically unavailable beyond final answer correctness. We propose Weakly Supervised GRPO (WS-GRPO), which improves rollout efficiency by converting terminal rewards into correctness-aware guidance over partial trajectories. Unlike global length penalties that are hard to calibrate, WS-GRPO trains a preference model from outcome-only correctness to produce prefix-level signals that indicate when additional continuation is beneficial. Thus, WS-GRPO supplies outcome-derived continue/stop guidance, reducing redundant deliberation while maintaining accuracy. We provide theoretical results and empirically show on reasoning benchmarks that WS-GRPO substantially reduces rollout length while remaining competitive with GRPO baselines.
Figures & tables
Figure 1: WS-GRPO for efficient reasoning. GRPO’s group-relative objective can favor longer trajectories, while simple length penalties are hard to calibrate and human supervision is expensive (top). WS-GRPO uses a preference model trained on correct vs. incorrect outcomes to compare consecutive prefixes, generating correctness-aware guidance that reduces redundant continuation while preserving necessary reasoning (bottom). Some icons were generated by a generative AI tool (ChatGPT) and are for illustrative purposes only.
Figure 2: WS-GRPO Framework Overview: Phase 1: using outcome-only correctness (final-answer labels), we construct preference pairs from correct vs. incorrect rollouts and train a preference model Pω . Phase 2: the frozen Pω converts terminal correctness into correctness-aware, prefix-level rewards by comparing consecutive prefixes within each rollout. These dense prefix-level rewards are combined with the terminal reward and used in the GRPO objective to refine the policy, improving rollout efficiency.
Qwen2.5-7B-Instruct
Qwen3-4B-Instruct
Dataset
Metric
GRPO
DRGRPO
WS-GRPO
Improve
GRPO
DRGRPO
WS-GRPO
Improve
ARC
Pass@1 ↑
0.904
0.904
0.879
Δ=−0.025
0.930
0.926
0.886
Δ=−0.044
Eval length (tok.) ↓
309.30
226.11
16.00
↓92.9%
161.94
268.47
20.23
↓87.5%
Avg steps ↓
14.72
11.51
2.00
↓82.6%
8.04
12.34
2.00
↓75.1%
CommonsenseQA
Pass@1 ↑
0.841
0.858
0.837
Δ=−0.021
0.774
0.797
0.768
Δ=−0.029
Eval length (tok.) ↓
63.25
62.09
50.76
↓18.2%
14.04
220.59
42.64
↑204%
Table 1: Main results (accuracy + efficiency) for GRPO vs DRGRPO vs WS-GRPO. We report test-set Pass@1 accuracy (↑) and test-set average reasoning steps (↓). Efficiency is additionally reported as the mean completion length in tokens (↓) measured during evaluation at the final validation checkpoint . Improve compares WS-GRPO against the best baseline among {GRPO, DRGRPO} for each metric (max for Pass@1; min for length/steps). Green cells indicate improvements in cost metrics (lower steps/length).
Figure 3: Figure on the left demonstrates the Trajectory Length Distribution of samples used in Analysis from GSM8K, middle figure shows the Absolute Score Difference over steps, on the right is the Average combined reward score as a function of trajectory length.
Figure 4: Validation step-efficiency (Pass@1 / average reasoning steps) across training for Qwen models. Higher values indicate greater accuracy per reasoning step. WS-GRPO consistently achieves higher step-efficiency.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Validation step-efficiency (Pass@1 / average reasoning steps) across training for Llama models. Higher values indicate greater accuracy per reasoning step.
Dataset
Training
Validation
Testing
ARC
1813
129
648
CommonsenseQA
2000
200
2740
Deepmath
2000
200
2883
GSM8K
2000
200
2198
Appendix
Table 2: Training/Validation/Testing Split for ARC and CommonsenseQA datasets.
Llama-3.2-3B-Instruct
Llama-3.1-8B-Instruct
Dataset
Metric
GRPO
DRGRPO
WS-GRPO
Improve
GRPO
DRGRPO
WS-GRPO
Improve
ARC
Pass@1 ↑
0.794
0.782
0.748
Δ=−0.046
0.816
0.841
0.824
Δ=−0.017
Eval length (tok.) ↓
137.20
97.91
14.00
↓85.7%
13.05
8.00
20.00
↑150%
Avg steps ↓
5.07
4.22
2.04
↓51.7%
1.00
1.00
2.00
↑100%
CommonsenseQA
Pass@1 ↑
0.714
0.717
0.688
Δ=−0.029
0.737
0.749
0.721
Δ=−0.028
Eval length (tok.) ↓
38.20
51.60
37.49
↓1.9%
165.03
441.09
38.44
↓76.7%
Appendix
Table 3: Main results (accuracy + efficiency) for GRPO vs DRGRPO vs WS-GRPO. We report test-set Pass@1 accuracy (↑) and test-set average reasoning steps (↓). Efficiency is additionally reported as the mean completion length in tokens (↓) measured during evaluation at the final validation checkpoint . Improve compares WS-GRPO against the best baseline among {GRPO, DRGRPO} for each metric (max for Pass@1; min for length/steps). Green cells indicate improvements in cost metrics (lower steps/length).
Group Relative Policy Optimization (GRPO) is widely used for training reasoning models, but updating all sampled completions in each group incurs substantial cost and can reinforce verbose reasoning trajectories. In this paper, we study whether all completions provide equally useful update signals in GRPO-style reasoning RL. Our gradient-similarity analysis shows that, within the same prompt group, same-class completions often induce highly similar update directions, whereas correct-incorrect pairs provide more distinct contrastive signals. Motivated by this observation, we propose Binary Prefix Policy Optimization (BPPO), which uses the shortest correct completion and the shortest incorrect completion as a compact update unit while preserving full-group advantage normalization. BPPO further improves efficiency with adaptive completion scheduling and prefix-focused optimization; by updating only response prefixes, it avoids reinforcing redundant suffixes and encourages more concise responses. Experiments on GSM8K, MATH, and Geo3K show that BPPO achieves up to 6.08x speedup over GRPO while maintaining competitive accuracy, and reduces mean response length by approximately 30-50% without modifying the reward with an explicit length penalty.
Reinforcement learning is now central to eliciting reasoning in large language models, while in the popular algorithm Group Relative Policy Optimization (GRPO) every token in a rollout receives the same advantage. We ask how to make process supervision efficient: accelerating convergence and improving final quality without the cost of value networks. We propose Bag-of-Tokens Group Relative Policy Optimization (BoT-GRPO), which extends GRPO to token-level reward models through a length-invariant "bag of tokens" aggregation: it collects all token-level rewards across rollouts, weights each by the inverse of its source sequence length, and computes per-token advantages relative to weighted group statistics. BoT-GRPO is critic-free, and is a drop-in replacement wherever GRPO is used when token-level reward is available. On React front-end code generation, BoT-GRPO reaches 80% compile rate up to 1.9× faster than GRPO and converges faster than modern GRPO variants (GSPO, DAPO, PURE) while reaching higher final compile and VLM-judged win rates. On a second task, AIME mathematical reasoning, BoT-GRPO delivers absolute Pass@k gains up to 8.1% over GRPO in half the steps. For both tasks we compare the algorithm's performance on reasoning vs. non-reasoning base-model families (Qwen2.5-3B, SmolLM3-3B, Phi-4-mini-reasoning). Our experiments also yield a practical recipe for the reward model itself: reward stability matters more than richness: clean, bounded, stable fine-grained signals consistently accelerate learning where noisier alternatives stall.
Group Relative Policy Optimization (GRPO) is an effective recipe for training reasoning models with verifier-based outcome rewards, but its supervision is sparse: when all sampled trajectories for a prompt receive the same verifier reward, the group-relative advantage collapses to zero and learning stalls. Outcome-only rewards also provide no step-level credit assignment, limiting exploration and making it harder to learn robust reasoning. We present VeriGate (Verifier-Gated Step-Level GRPO), a verifier-gated extension of GRPO that addresses these limitations with three design choices. First, VeriGate keeps the verifier in charge whenever verifier rewards induce a meaningful preference among sampled trajectories, and uses process supervision only when verifier rewards are degenerate. Second, instead of collapsing Process Reward Model (PRM) step scores into a single trajectory reward, VeriGate converts them into future-cumulated rewards to assign continuation-aware credit. Third, VeriGate transforms these rewards into group-normalized token-level advantages, restoring informative gradients and fine-grained credit assignment while remaining less susceptible to reward hacking than methods that optimize aggregated PRM scores. Empirically, training on MATH with 1.5B and 7B Qwen2.5-Instruct models and evaluating on six reasoning benchmarks, VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively, substantially reduces zero-gradient failures, decreases reward-hacking behavior, and improves reasoning quality relative to outcome-only GRPO and PRM-as-outcome baselines.