Process Rewards for Outcome-Guided Reasoning Steps in LLM Reinforcement Learning

Mohammad Rezaei; Jens Lehmann; Sahar Vahdati

cs.LGFeb 8, 2026

Process Rewards for Outcome-Guided Reasoning Steps in LLM Reinforcement Learning

Authors: Mohammad Rezaei, Jens Lehmann, Sahar Vahdati

Organizations: Faculty of Computer Science, Technical University of Dresden, Dresden, Germany · Amazon Science · Technical University of Dresden, Dresden, Germany · TIB Leibniz Information Centre, Leibniz University of Hanover, Hanover, Germany

Abstract

Mathematical reasoning in large language models has improved substantially with reinforcement learning using verifiable rewards, where final answers can be checked automatically and converted into reliable training signals. Most such pipelines optimize outcome correctness only, which yields sparse feedback for long, multi-step solutions and offers limited guidance on intermediate reasoning errors. Recent work therefore introduces process reward models (PRMs) to score intermediate steps and provide denser supervision. In practice, PRM scores are often imperfectly aligned with final correctness and can reward locally fluent reasoning that still ends in an incorrect answer. When optimized as absolute rewards, such signals can amplify fluent failure modes and induce reward hacking. We propose PROGRS, a framework that leverages PRMs while keeping outcome correctness dominant. PROGRS treats process rewards as relative preferences within outcome groups rather than absolute targets. We introduce outcome-conditioned centering, which shifts PRM scores of incorrect trajectories to have zero mean within each prompt group. It removes systematic bias while preserving informative rankings. PROGRS combines a frozen quantile-regression PRM with a multi-scale coherence evaluator. We integrate the resulting centered process bonus into Group Relative Policy Optimization (GRPO) without auxiliary objectives or additional trainable components. Across MATH-500, AMC, AIME, MinervaMath, and OlympiadBench, PROGRS consistently improves Pass@1 over outcome-only baselines and achieves stronger performance with fewer rollouts. These results show that outcome-conditioned centering enables safe and effective use of process rewards for mathematical reasoning.

Figures & tables

Metric	Model / Setting	AIME 24	AIME 25	AMC 23	MATH-500	MinervaMath	OlympiadBench
Pass@1 (%)	DAPO-16	$8.67\pm 4.00$	$8.00\pm 1.63$	$52.00\pm 3.67$	$69.66\pm 0.70$	$18.79\pm 1.47$	$32.03\pm 0.73$
	DAPO-8	$8.67\pm 1.63$	$\mathbf{12.33\pm 2.13}$	$48.75\pm 5.62$	$68.40\pm 0.80$	$14.63\pm 0.59$	$31.34\pm 0.87$
	PROGRS-4	$10.67\pm 2.00$	$12.00\pm 1.63$	$50.25\pm 2.61$	$74.14\pm 0.98$	$\mathbf{23.64\pm 0.66}$	$\mathbf{35.74\pm 0.89}$
	PROGRS-8	$\mathbf{12.00\pm 3.06}$	$10.67\pm 2.91$	$\mathbf{59.00\pm 5.27}$	$\mathbf{74.92\pm 0.82}$	$22.13\pm 0.49$	$35.06\pm 0.61$
	Ablation: $\alpha_{\text{coh}}=0$	$9.33\pm 3.59$	$9.67\pm 2.33$	$50.50\pm 2.18$	$71.82\pm 1.07$	$20.18\pm 0.80$	$33.52\pm 0.58$
	Ablation: No Centering	$12.00\pm 3.06$	$8.33\pm 1.67$	$54.00\pm 3.74$	$67.78\pm 0.95$	$20.15\pm 0.75$	$30.77\pm 0.64$

TABLE I : Results and ablations across mathematical reasoning benchmarks. We report Pass@1 accuracy (upper block, in %) and average generated tokens per problem (lower block; lower is better), both as mean

\pm

standard deviation over ten independent runs. DAPO-

K

denotes the outcome-only baseline trained with

K

rollouts per prompt; PROGRS-

K

denotes our method with the same rollout budget. Ablation rows remove individual components while keeping all other settings fixed:

\alpha_{\text{coh}}=0

disables the coherence penalty (centering retained), and No Centering removes outcome-conditioned centering (coherence retained).

Fig. 1 : Computational efficiency across models. (a) Pareto frontier: Accuracy vs. budget; PROGRS-4 matches DAPO baselines with

\sim 50\%

budget, PROGRS-8 matches/exceeds DAPO-16. (b) Efficiency ranking: Accuracy per token; PROGRS-8 highest, PROGRS-4 next. (c) Efficiency score: Accuracy normalized by budget; PROGRS-4 best. (d) Efficiency-budget surface: PROGRS-4 optimal for

\leq 50\%

budget, PROGRS-8 for

50\%-100\%

Fig. 2 : Pass@K accuracy across benchmarks. Each panel shows Pass@1, 5, 10 for DAPO-16 (red), DAPO-8 (orange), PROGRS-4 (green), and PROGRS-8 (blue). Shaded regions indicate standard deviation across evaluation runs.

Fig. 3 : Training dynamics of DAPO vs. PROGRS: mean and std. of advantages and per-token entropy over training (smoothed with a 50-step moving average).

Dataset	PROGRS Avg@8	PROGRS Avg@16	PROF-GRPO Avg@16	PRL Avg@8
AMC23	58.75	60.47	49.1	45.94
MATH-500	75.42	74.82	73.2	71.00
Minerva	22.15	22.36	30.0	28.63
Olympiad	37.91	38.02	36.1	33.57
AIME24	10.83	10.21	9.6	9.17

TABLE II : Comparison with prior PRM-based methods under a shared base model. Avg@8/16 (%).

	Model	AIME24	AIME25	AMC23	MATH-500	Minerva	Olympiad
Pass@1 (%)	Global Centering	13.3 $\pm$ 2.6	3.0 $\pm$ 3.1	50.8 $\pm$ 3.5	63.8 $\pm$ 0.7	14.1 $\pm$ 0.6	30.4 $\pm$ 1.0
	Rank Shaping	3.0 $\pm$ 2.3	8.3 $\pm$ 1.7	35.0 $\pm$ 4.0	60.9 $\pm$ 1.0	12.2 $\pm$ 0.6	24.4 $\pm$ 1.0
	Step Shaping	5.7 $\pm$ 3.4	4.3 $\pm$ 3.7	44.0 $\pm$ 3.0	65.7 $\pm$ 0.9	18.3 $\pm$ 0.7	28.9 $\pm$ 0.8
	PRM Filtering	6.7 $\pm$ 2.6	3.0 $\pm$ 1.8	47.0 $\pm$ 4.2	63.0 $\pm$ 0.7	15.7 $\pm$ 0.7	27.4 $\pm$ 1.1
	PROGRS-4	10.7 $\pm$ 2.0	12.0 $\pm$ 1.6	50.3 $\pm$ 2.6	74.1 $\pm$ 1.0	23.6 $\pm$ 0.7	35.7 $\pm$ 0.9

TABLE III : Rollout-matched baselines (

K=4

). Pass@1 (%, higher better), mean

\pm

std over 10 runs.

	Model	AIME24	AIME25	AMC23	MATH-500	Minerva	Olympiad
Pass@1 (%)	DAPO16-7B	24.0 $\pm$ 1.3	13.3 $\pm$ 2.1	66.0 $\pm$ 4.1	77.3 $\pm$ 0.5	36.0 $\pm$ 0.9	38.8 $\pm$ 0.9
	DAPO8-7B	20.0 $\pm$ 2.1	10.7 $\pm$ 4.4	58.0 $\pm$ 6.0	68.2 $\pm$ 1.1	26.0 $\pm$ 1.2	34.5 $\pm$ 0.6
	PROGRS4-7B	19.3 $\pm$ 1.3	10.7 $\pm$ 4.9	53.0 $\pm$ 1.9	70.9 $\pm$ 0.9	29.2 $\pm$ 0.4	36.3 $\pm$ 0.6
	PROGRS8-7B	22.0 $\pm$ 2.7	12.7 $\pm$ 2.5	57.0 $\pm$ 1.9	81.5 $\pm$ 0.5	37.5 $\pm$ 0.5	41.6 $\pm$ 0.6

TABLE IV : Scaling to a 7B policy. Pass@1 (%), mean

\pm

std over 10 runs.

Fig. 4 : Effect of outcome-conditioned centering on incorrect trajectories. Left: histogram of raw PRM scores vs. centered scores for

r=0

samples. Right: variance comparison; centering reduces variance by

75.9\%

while enforcing zero mean on incorrect bonuses (

\mathbb{E}[S^{\text{PRM}}|r=0]\approx 0.558

\operatorname{Var}[S^{\text{PRM}}|r=0]\approx 0.081

;

\mathbb{E}[\tilde{S}|r=0]=0

\operatorname{Var}[\tilde{S}|r=0]\approx 0.020

Explore similar work

Jul 3, 2026

Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising paradigm for improving mathematical reasoning in language models. Yet most RLVR work rewards only the final answer (outcome-based rewards), leaving the impact of step-level process supervision (process rewards) underexplored especially for small models that lack the capacity to self-correct under sparse feedback. We systematically compare five reward conditions applied to Qwen2.5-0.5B fine-tuned with Group Relative Policy Optimization (GRPO) on GSM8K: a no-RL baseline, process-only, outcome-only, and three hybrid weightings (

λ\in \{0.9, 0.5, 0.1\}

process weight). Process-only supervision achieves 63.73% test accuracy versus 53.75% for outcome-only, a nearly 10-percentage point gap while yielding reasoning traces with higher step validity and lower deviation from ground-truth chain length. Hybrid rewards generally correlate positively with process weight, with one notable anomaly: the low-process / high-outcome configuration (

λ=0.1

) underperforms pure outcome supervision, suggesting conflicting optimization signals. Error analysis using GPT-4o as a judge reveals distinct failure mode distributions: process models generate structurally inconsistent but arithmetically grounded traces, while outcome models produce concise but derivation-error-prone chains. Our results demonstrate that reward granularity is a first-order design decision for RLVR, with process-level supervision substantially improving both accuracy and trace fidelity in small language models.

Anagha Radhakrishna Palandye, Rebecca Glick, Osheen Kaul

New York University

Process Reward Models RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Ask PDF

Dec 2, 2025

ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth

Training process reward models (PRMs) requires step-level correctness labels, obtained either through expensive human annotation or by relying on ground-truth answers, limiting the ability to scale process-level supervision. We propose ScalePRM, which scales verification compute as an alternative: given a problem and a candidate solution, we generate multiple independent verifications of each reasoning step and aggregate their judgments to produce synthetic step-level labels without ground truth. We explore two representative inference-time scaling strategies, parallel scaling through self-consistency and sequential scaling through meta-critique, and train generative PRMs on the resulting synthetic data. On ProcessBench, a benchmark for identifying erroneous steps in mathematical reasoning, PRMs trained on step-level self-consistency data achieve 67.5 F1, surpassing reference-guided training with ground-truth access (66.4 F1) and GPT-4o as a critic (61.9 F1). When deployed as reward signals in RL training with Qwen2.5-Math-7B, our best PRM achieves 47.4% average accuracy across six mathematical reasoning benchmarks, outperforming ground-truth-based RLVR (43.9%). We also identify and address reward exploitation patterns unique to generative PRM-based RL. Our results demonstrate that scaling verification compute is a viable alternative to ground-truth supervision for training process reward models.

Salman Rahman, Sruthi Gorantla, Arpit Gupta +3

2UCLA · Work done while as an intern at Amazon AGI. · 1Amazon AGI

Process Reward Models Reinforcement Learning with Verifiable Rewards LLM Mathematical Reasoning

Ask PDF

May 17, 2026

Step-wise Rubric Rewards for LLM Reasoning

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models, but rewards only final-answer correctness with no supervision over intermediate steps. Rubric-based methods such as Rubrics as Rewards (RaR) introduce finer-grained supervision by scoring rollouts against structured criteria, yet the rubric scores are still aggregated into a single scalar applied to the entire response, causing three weaknesses: loss of multi-criterion structure, uniform supervision of correct and incorrect steps, and reward hacking through unbounded self-correction. On 1,000 problems, we find 18.2% of steps in correct-answer responses are wrong yet positively rewarded, while 49.9% of steps in incorrect-answer responses are correct yet penalized. We introduce Step-wise Rubrics as Rewards (SRaR), an RLVR framework that (i) uses an LLM judge to attribute each rubric item to a specific reasoning step, (ii) normalizes per-step rubric scores across rollouts so only steps whose quality varies produce a learning signal, and (iii) combines the per-step reward with the outcome reward through a decoupled advantage estimator that keeps the outcome baseline stable. We further build a 16K-problem rubric dataset by contrastively distilling rubric items from correct and flawed reasoning paths sampled from a strong model. Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points on Qwen3-8B and 2.75 points on Qwen3-32B, raises the Faithful Reasoning Rate on AIME 2025 from 34.5% to 46.7%, and reduces self-correction looping from 48.1% to 26.5%.

Weichu Xie, Haozhe Zhao, Wenpu Liu +15

Peking University · JD Explore Academy · Shanghai Jiao Tong University +2

Reward Modeling Credit Assignment in RL Step-Level Credit Assignment in RL Structured Reasoning Rubric-Based RL Reinforcement Learning with Verifiable Rewards+4

Ask PDF