ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning

Diyi Hu; Ruizhong Qiu; Lin Qiu; Hanghang Tong; Hanqing Zeng

cs.LGOct 17, 2026

ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning

Authors: Diyi Hu, Ruizhong Qiu, Lin Qiu, Hanghang Tong, Hanqing Zeng

Organizations: Independent Researcher · University of Illinois Urbana-Champaign · Meta

Abstract

Reinforcement learning from verifiable rewards assigns one terminal reward to an entire reasoning trajectory, obscuring how individual reasoning steps contributed to the outcome. GRPO broadcasts a group-normalized advantage to every response token, while finer-grained approaches often rely on dense auxiliary rollouts or branching schemes that can also alter the data used for policy updates. We cast the missing middle as budgeted measurement: with a limited rollout budget, which partial solutions should be evaluated, and how should their value estimates refine credit? ProbePO selects partial-solution states with a reward-free uncertainty score and estimates their expected verifier rewards under a capped continuation budget. ProbePO redistributes, rather than replaces, GRPO's trajectory-level advantage across tokens while preserving each trajectory's mean token-level advantage; probe continuations serve only as measurements. Across three DeepSeekMath and Qwen-family models spanning 1.5B-7B parameters, ProbePO consistently outperforms GRPO and achieves the highest accuracy among the four evaluated methods in 16 of 18 model-benchmark comparisons. Against dense-probing VinePPO on DeepSeekMath-7B, ProbePO achieves higher accuracy on all six benchmarks with 76% less end-to-end training time.

Figures & tables

Panel of Figure 1 — Figure 1: ProbePO improves cost-quality tradeoff by assigning within-trajectory credit with a sparse-probe budget. GRPO incurs no auxiliary-probe overhead but provides only trajectory-level credit. Dense per-boundary probing, exemplified by VinePPO, provides fine resolution at high rollout cost, while finite-sample Monte Carlo differences remain noise-sensitive. ProbePO instead caps the auxiliary-probe budget, concentrates it on selected pivots, and shrinks their value estimates before deriving segment-wise credit. The pipeline first fixes the policy-update trajectories, then ranks and probes candidate boundaries under the group-wide budget, and finally adds a token-mean-zero segment residual to the GRPO base advantage.

Method	MATH-500	Olympiad Bench	College Math	AMC	Minerva Math	AIME24 (@32)	Avg-6
DeepSeekMath-7B-SFT
GRPO	42.55	12.67	22.57	14.38	11.79	1.77	17.62
SPO-tree	37.58	9.93	20.11	12.65	10.96	0.83	15.34
TreePO	43.86	11.73	22.46	11.22	9.38	0.83	16.58
ProbePO (ours)	45.81	14.47	22.77	17.09	12.96	2.92	19.34
R1-Distill-Qwen-1.5B

Table 1: Terminal accuracy (%) across three backbones and six mathematical reasoning benchmarks. Avg-6 is the unweighted benchmark mean; bold marks the best score within each backbone.

Figure 2: Boundary values and segment credit. Two held-out MATH responses from the trained R1-Distill-Qwen-1.5B checkpoint: (a) a correct cube-path response and (b) an incorrect parallelogram response. Top panels show the anchor, shrunk pivot estimates (

M=8,n_{0}=4

), and terminal reward; bottom panels compare ProbePO and GRPO advantages on a shared scale, with widths proportional to token counts. The detour (

I_{2}

, a) receives negative credit; the answer reversal (

I_{3}

, b) receives the strongest penalty. Each ProbePO profile preserves the GRPO token mean; in (b),

r=0.10

denotes a boxed but incorrect answer. Appendix D.5 provides the full contexts and pooled-

M=32

check.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Terminal accuracy (%; higher is better)
Method	MATH 500	Olympiad Bench	College Math	AMC	Minerva Math	AIME24 (@32)	Avg-6
ProbePO (ours)	45.81	14.47	22.77	17.09	12.96	2.92	19.34
VinePPO	43.69	11.65	22.11	15.81	10.27	0.31	17.31

Appendix

Table 3: Quality and cost against dense probing. Panel (a) reports terminal benchmark accuracy under the shared DeepSeekMath setting; panel (b) reports end-to-end training cost.

Method	Training time (hours)	MATH-500	Avg-6
ProbePO	11.14	80.01	41.03
GRPO	11.14	75.52	39.08
$\Delta$	—	$+4.49$	$+1.95$

Appendix

Table 4: Equal-training-time Qwen3-4B comparison (%). Both methods use approximately 11.14 logged trainer hours on the same

8\times

B200 hardware.

Figure 5: Qwen3-4B training dynamics. Monitoring accuracy over the shared headline horizon; points denote evaluations and lines connect successive checkpoints.

Profile	Segment 1	Segment 2	Segment 3
Panel (a), deployed $M=8$	$+0.34$	$-0.41$	$+0.72$
Panel (a), pooled $M=32$	$+0.41$	$-0.10$	$+0.53$
Panel (b), deployed $M=8$	$-1.08$	$-1.08$	$-2.89$
Panel (b), pooled $M=32$	$-1.37$	$-0.80$	$-3.24$

Appendix

Table 5: Sensitivity of the displayed token-advantage profiles. The pooled-

M=32

calculation preserves the

(+,-,+)

pattern in column (a) and the terminal-most-negative ordering in column (b).

(a) Measurement and value estimation
Method	Budgeted spend	Reward-free interior pilot	Estimates $V^{\pi_{\theta}}(s)$
GRPO [ Shao et al., 2024 ]	(no spend)	—	—
VinePPO [ Kazemnejad et al., 2025 ]	$\times$	— (uniform)	✓
SPO-tree [ Guo et al., 2025 ]	$\times$	$\times$	$\pm$
TreeRPO [ Yang et al., 2025b ]	$\times$	$\times$	$\pm$
TreePO [ Li et al., 2026a ]	$\pm$	$\times$	$\pm$

Appendix

Table 6: Mechanism comparison. Cells summarize the cited configurations.

Panel of Figure 6 — Figure 6: Offline selector diagnostics. (a) Rank correlation between reward-free signals and action-fork variance. (b) Oracle variance captured by the top-ranked versus a uniformly selected boundary. Error bars and brackets show the bootstrap intervals described in the text.

Parameter	Value
Backbone group size $G$	8 responses per prompt
Group-wide probe budget $B$	40 continuation rollouts
Continuations per selected pivot $M$	8
Shrinkage pseudo-count $n_{0}$	4
Residual weight $\alpha$	1
Stage-1 score	$H_{\max}$ with forward window $L=8$

Appendix

Table 7: Shared ProbePO acquisition and estimator settings. These values apply to the three headline ProbePO arms in Table 1 .

Parameter	DeepSeekMath 7B	R1-Distill-Qwen 1.5B (2K context)	Qwen3 4B
Learning rate	$5{\times}10^{-7}$	$1{\times}10^{-6}$	$1{\times}10^{-6}$
Schedule	constant; no warmup	3% linear warmup, then constant	constant; no warmup
Batch / mini	$64/8$	$64/16$	$512/8$
PPO epochs	1	1	1
PPO clip ratio	0.2	0.2	0.2
KL regularization	loss-side $10^{-3}$ low_var_kl	loss-side $10^{-3}$ low_var_kl	loss-side $10^{-3}$ low_var_kl

Appendix

Table 8: Optimization and batching for the headline ProbePO runs. Batch and mini-batch are prompt counts.

Parameter	DeepSeekMath 7B	R1-Distill-Qwen 1.5B (2K context)	Qwen3 4B
Train/probe temperature, top- $p$	$0.8,\ 0.95$	$1.0,\ 1.0$	$1.0,\ 1.0$
Maximum prompt tokens	512	512	2,048
Configured response-token limit	1,024	4,096	4,096
Rollout context limit	4,096	2,048 (context binding)	6,144
Training horizon (steps)	1,000	1,000	27

Appendix

Table 9: Decoding, context, and training horizons for the headline ProbePO runs. Probe continuations inherit the backbone decoding parameters.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 435 papers

Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts
Mar 22, 2026Andrei Baroian, Rutger BergerPrompt Selection for RL RL Post-Training
Leiden Institute of Advanced Computer Science Leiden University
VeriGate: Verifier-Gated Step-Level Supervision for GRPO
May 28, 2026Aakriti Agrawal, Minghui Liu, Furong HuangProcess Reward Models RL for Language Model Reasoning
University of Maryland, College Park
GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
Apr 22, 2026Jingyi Wang, Lei Zhu, Tengjin Weng +8RL for Language Model Reasoning Group Relative Policy Optimization
Tsinghua University · Huawei Noah’s Ark Lab · Shenzhen University

Mar 22, 2026J/K move · Enter open · S save

Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts

Andrei Baroian, Rutger Berger

Leiden Institute of Advanced Computer Science Leiden University

Prompt Selection for RL RL Post-Training Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) plays a crucial role in expanding the capacities of LLM reasoning, but GRPO-style training is dominated by expensive rollouts and wastes compute on unusable prompts. We propose Prompt Replay, an overhead-free online data selection method for GRPO that reuses prompts only (not trajectories), to preserve on-policy optimization. After each step, we insert prompts with medium difficulty into a buffer, and prioritize prompts closer to a pass rate of 0.5 (half answers correct, half wrong) to maximize the advantage, thus learning signal. Training batches are formed by mixing reused prompts with fresh samples, with cooldown steps and max reuse times controlling aggressiveness vs risk of overfitting. Across multiple model families (Llama-3.2- 3B, Qwen3-8B) and training datasets (Dolci, Polaris), evaluated using average accuracy on six standard math benchmarks, Prompt Replay reduces zero-variance prompts, increases mean absolute advantage and shows faster initial accuracy gains. Yet, it plateaus and converges with the baseline, as too aggressive configuration was used. The method is most efficient when the rollouts are the primary bottleneck and the dataset is difficult for the model. We additionally observe that Qwen2.5-Math can exhibit spurious-reward effects that invalidates ablations, raising a warning signal for using it as a sole testbed for GRPO method research.

Abstract

RL for Language Models

Explore similar work

Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts

VeriGate: Verifier-Gated Step-Level Supervision for GRPO

GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning