RAWR: Reward Assignment Without Rollouts in Verifiable Domains

Corentin Royer; Anna Hedström; Debarun Bhattacharjya; Gaetano Rossiello; Andrea Giovannini; Mennatallah El-Assady

cs.CLMar 18, 2026

RAWR: Reward Assignment Without Rollouts in Verifiable Domains

Authors: Corentin Royer, Anna Hedström, Debarun Bhattacharjya, Gaetano Rossiello, Andrea Giovannini, Mennatallah El-Assady

Organizations: International Business Machines · Department of Computer Science, ETH Zurich, 8092 Zurich, Switzerland · ETH AI Center · Lirio · Department of Computer Science, ETH Zurich

Abstract

Understanding and evaluating multi-step reasoning in LLMs at the level of individual steps remains a key challenge. Process reward models (PRMs) provide a solution by scoring each step, enabling fine-grained supervision and improved reliability. However, training them requires costly human annotation or computationally intensive rollout-based labeling. To solve this, we introduce MCNIG, a scalable method for automatically labeling the quality of individual reasoning steps in any verifiable domain. Its step score, net information gain (NetIG), improves upon single-reference information gain (IG) by comparing the most-supported correct answer against the most-supported incorrect one, yielding a robust signal even for long and structured outputs like code and SQL, where IG fails. We show that the signal produced by MCNIG correlates with human judgments of step quality, and we apply MCNIG labels to train PRMs that achieve the best average best-of-K accuracy across eight benchmarks spanning mathematics, code generation, text-to-SQL, and scientific QA. Crucially, MCNIG generates no rollouts, cutting labeling complexity to O(N) and making it up to X times cheaper than rollout-based methods at comparable label quality, which makes large-scale process supervision practical.

Figures & tables

Figure 1: Overview of RAWR. (1) An LLM samples

K

CoTs for question

q

. (2) A verifier

\mathcal{V}_{q}

partitions them into correct (

\mathcal{C}_{q}

) and incorrect (

\mathcal{W}_{q}

) sets. (3) At every prefix

r_{0},\dots,r_{n}

, we score each candidate answer and keep the

\max

over

\mathcal{C}_{q}

and

\mathcal{W}_{q}

. (4) The step score, NetIG, is the change in the correct-incorrect gap relative to

r_{0}

, thresholded at

\tau

to produce binary step labels.

Method	Label type	Complexity	Rollout-free	Cross-domain
PRM800K ( Lightman et al., 2023 )	Human expert	—	✓	✗
MathShepherd ( Wang et al., 2024a )	Rollout	$\mathcal{O}(N^{2})$	✗	✗
OmegaPRM ( Luo et al., 2024 )	Rollout + binary search	$\mathcal{O}(N\log N)$	✗	✗
QwenPRM ( Zhang et al., 2025 )	Rollout + consensus filtering	$\mathcal{O}(N^{2})$	✗	✗
RAWR (ours)	Information Theoretic	$\mathcal{O}(N)$	✓	✓

Table 1: Comparison of step-level supervision methods. RAWR requires no rollouts and achieves

\mathcal{O}(N)

complexity. Relevant prior methods have been validated on mathematics only.

Figure 2: Left: Balanced accuracy of CoT -level labels for IG vs. RAWR across domains. Right: Accuracy stratified by the number of distinct correct answer formulations. IG degrades as formulation diversity increases, while RAWR remains stable.

Figure 3: ProcessBench Step F1 against labeling cost relative to RAWR ( Section A.1 ); top-left is best. Filtered QwenPRM adds an LLM-as-a-judge filtering stage to the same rollouts, raising F1 but discarding half of the labeled samples ( $155\times$ ).

Figure 4: Best-of-

K

accuracy as a function of

K

, stratified by ground-truth answer length, for PRMs trained on IG labels and on RAWR labels. The RAWR- IG gap is negligible for short answers and widens as answers grow longer, confirming that single-reference IG degrades on structured, long-form outputs while RAWR remains robust. Shaded bands show

\pm 1

standard deviation across repeated subsamples of

K

answers per problem.

Method	MATH	GSM	PubMed	HumanEval	BigCode	BIRD	AIME	Average
Single sampling	44.5%	73.3%	36.2%	55.5%	23.3%	19.6%	17.4%	38.5%
Majority voting	67.3%	91.2%	46.8%	66.1%	27.6%	37.6%	29.0%	52.2%
OVM	62.8%	92.0%	59.4%	76.3%	34.9%	45.1%	26.1%	56.6%
MathShepherd	56.1%	89.5%	49.7%	52.5%	29.5%	35.9%	23.0%	48.0%
ImplicitPRM	59.3%	89.9%	44.3%	77.8%	29.3%	35.6%	19.6%	50.8%
Granite PRM	62.5%	90.9%	49.8%	61.6%	31.4%	35.4%	27.0%	51.2%

Table 2: Best-of-

K

accuracy (

K{=}32

) across eight benchmarks. Bold indicates the best result and underline the second best, computed separately within the 8B group and globally for RAWR 14B. Shaded rows are our methods; RAWR 8B achieves the highest average performance among 8B models.

Method	UGPhysics	ProcessBench Trace	ProcessBench Step
Single sampling	8.0%	—	—
Majority voting	11.5%	—	—
OVM	9.2%	33.3	18.6
MathShepherd	10.0%	41.1	31.5
ImplicitPRM	10.6%	68.6	21.2
Granite PRM	11.0%	51.9	51.4

Table 3: External evaluation. UGPhysics : best-of-

K

accuracy (OOD domain). ProcessBench Trace: F1 for classifying whether a trace contains an error (introduced by us). ProcessBench Step: official F1 for localizing the first incorrect step. Bold indicates the best result and underline the second best, computed separately within the 8B group and globally for RAWR 14B.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Provider (API share)	Model	Input	Output	Output / input
Anthropic (40%)	Claude Fable 5.1	10.00	50.00	$5.0\times$
	Claude Opus 5.5	4.00	20.00	$5.0\times$
	Claude Sonnet 5	2.00	10.00	$5.0\times$
	Claude Haiku 4.5	1.00	5.00	$5.0\times$
OpenAI (27%)	GPT-6 Astra	10.00	50.00	$5.0\times$
	GPT-6 Sol	2.00	10.00	$5.0\times$

Appendix

Table 4: API list prices (USD per million tokens; standard tier, short context) of the three providers that together account for 88% of enterprise LLM API usage ( Tully et al., 2025 ) , accessed 26 September 2026 ( Anthropic, 2026 ; OpenAI, 2026 ; Google, 2026 ) . Every model bills output at

5

–

8.3\times

the input rate (median

5\times

), the multiplier used in Table 5 .

	Qwen PRM	MathShepherd	OmegaPRM	Granite PRM	RAWR
Input tokens	$1.8\times 10^{8}$	$0.44\times 10^{8}$	$0.44\times 10^{8}$	$0.44\times 10^{8}$	$\mathbf{1.5\times 10^{8}}$
Output tokens	$45.8\times 10^{8}$	$22.9\times 10^{8}$	$6.6\times 10^{8}$	$6.6\times 10^{8}$	$\mathbf{0}$
Cost-weighted (out $=5\times$ )	$230.6\times 10^{8}$	$114.9\times 10^{8}$	$33.6\times 10^{8}$	$33.6\times 10^{8}$	$\mathbf{1.5\times 10^{8}}$
relative to RAWR	$155\times$	$77\times$	$23\times$	$23\times$	$\mathbf{1\times}$
relative to RAWR, raw tokens (out $=1\times$ )	$32\times$	$15.7\times$	$4.8\times$	$4.8\times$	$\mathbf{1\times}$

Appendix

Table 5: Tokens processed by each labeling method to produce the same number of labeled samples on PRM800K , split into input (prefill/scoring) and output (generation), after normalizing by each method’s retention rate

\rho

. CoT generation is shared across all methods and excluded. Rollout-based methods spend

94

–

98\%

of their tokens on output; RAWR generates none. The cost-weighted total prices output at

5\times

input, the median list-price ratio of the major API providers ( Table 4 ); the last two rows give the cost relative to RAWR at that rate and, for reference, counting raw tokens (

1\times

). QwenPRM uses the MathShepherd rollouts plus a separate LLM-as-a-judge filtering stage; its judge verdict tokens are excluded, so its cost is a lower bound. Granite PRM uses Automatic Process Supervision (the OmegaPRM MCTS labeling method), so we estimate its labeling cost as equal to OmegaPRM ’s; this counts the labeling stage only, not its additional fine-tuning.

Dataset	# Problems	# Candidate answers
MATH500 ( Lightman et al., 2023 )	12,000	85,248
GSM8K ( Cobbe et al., 2021 )	8,790	31,768
MathQA ( Amini et al., 2019 )	29,837	204,974
AquaRat ( Ling et al., 2017 )	97,700	589,446
NumGLUE ( Mishra et al., 2022 )	41,018	265,532
PubMedQA ( Jin et al., 2019 )	1,000	3,727

Appendix

Table 6: Training dataset composition.

Dataset	Avg # step	BoK	% correct	BoK	% correct
		Train		Test
MATH500	8.50	86.57%	33.37%	88.98%	51.10%
GSM8K	7.52	98.55%	75.81%	97.72%	84.30%
MathQA	8.08	81.22%	38.52%	-	-
AquaRat	8.29	71.65%	28.19%	-	-
NumGLUE	5.80	86.05%	38.34%	-	-

Appendix

Table 7: Statistics of the generated CoTs. The BoK columns show the oracle best-of-

K

: the percentage of problems where at least one of the

K

samples is correct, whereas the % correct columns show the percentage of generations that are correct.

Figure 5: Distribution of the number of steps in the CoTs of all the datasets combined.

Figure 6: Average best-of-

K

performance over the evaluation tasks for PRMs trained on different subsets of the training set, ranging from 100K to the full 1.2M samples.

Aggregation	Baseline anchor ( $r_{0}$ )	Successive diff. ( $r_{i-1}$ )
max	82.01	80.46
sum	81.60	80.09

Appendix

Table 8: Balanced accuracy (%) of RAWR labels under each aggregation rule (

\max

, sum) and baseline anchor, micro-averaged across the 13 training datasets.

\max

outperforms sum under both anchors, and anchoring at the pre-reasoning baseline (

\text{NetInfo}_{0}

) outperforms the strictly local successive difference (

\text{NetInfo}_{i-1}

) under every aggregation rule.

$i/N$	$0.0$	$0.2$	$0.4$	$0.6$	$0.8$	$1.0$
Correct ( $n{=}484$ )	0.149	0.153	0.165	0.376	0.886	0.995
Incorrect ( $n{=}757$ )	0.140	0.133	0.113	0.107	0.069	0.027

Appendix

Table 9: Median length-normalized gold-answer probability

p_{i}(y^{\star})

against the normalized step index

i/N

, teacher-forced on 1,241 ProcessBench traces. Correct and incorrect traces are indistinguishable before reasoning and diverge monotonically as it accumulates.

	Balanced acc.	Traces
Full set	$83.0\%\pm 0.1$	85,248
High-prior ( $p_{0}>0.9$ )	$89.9\%\pm 0.8$	1,298

Appendix

Table 10: Balanced accuracy of RAWR labels on the full MATH training set versus the high-prior subset (

p_{0}>0.9

), same protocol as the main paper. Spreads are

\pm 1

std over a bootstrap. Labels on saturated priors are more accurate than on the full set.

Verifier noise $\varepsilon$	BIRD	$\Delta$ vs. clean	MATH	$\Delta$ vs. clean
$0.00$ (clean)	$85.34\%$	—	$81.59\%$	—
$0.01$	$84.41\%$	$-0.93$	$81.50\%$	$-0.09$
$0.05$	$82.22\%$	$-3.13$	$79.30\%$	$-2.29$
$0.10$	$80.15\%$	$-5.19$	$76.94\%$	$-4.65$

Appendix

Table 11: Balanced accuracy of RAWR labels under a verifier that mislabels each candidate with probability

\varepsilon

(7 seeds per level). Degradation is sub-linear and the labels remain usable across the full range tested.

$s$	5	10	20	30	40	512 (ref)
Balanced acc.	69.8%	72.7%	73.9%	74.2%	74.1%	77.9%
Gap to reference	8.1	5.2	4.0	3.7	3.8	—

Appendix

Table 12: Downstream balanced accuracy as a function of the number of subsampled candidates

s

(100 MATH problems, 512-candidate reference pool). Accuracy plateaus by

s\approx 20

30

; the paper’s

K{=}32

sits on the plateau.

Figure 7: Example of a sample that is misclassified under IG but correctly identified under RAWR, illustrating the sharper discriminative behavior of the max-based information signal. The right axis shows the NetIG score of RAWR.

Panel of Figure 8 — Figure 8: Error step offset distributions on ProcessBench . Left: QwenPRM 7B, tightly concentrated at offset 0, indicating precise step-level localization. Right: RAWR 8B, with the mode still at offset 0 but wider spread predominantly in $[-2,+2]$ . The right skew (about 56 traces at positive offsets vs. 36 at negative ones) indicates a tendency to flag errors later than ground truth.

GSM8K	MATH	Olympiad	OmniMath
82.8%	77.2%	81.5%	80.2%

Appendix

Table 13: Percentage of flagged incorrect ProcessBench solutions whose first flagged step precedes the final step. An outcome-level signal would place its flag at the last step, giving values near

0

Explore similar work

Step-wise Rubric Rewards for LLM Reasoning
May 17, 2026Weichu Xie, Haozhe Zhao, Wenpu Liu +15Reinforcement Learning With Verifiable Reward Rubric-Based Scoring
Peking University · JD Explore Academy · Shanghai Jiao Tong University +2
ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth
Dec 2, 2025Salman Rahman, Sruthi Gorantla, Arpit Gupta +3Process Reward Model Mathematical Reasoning Benchmarks
2UCLA · Work done while as an intern at Amazon AGI. · 1Amazon AGI
Rethinking Reward Models for Multi-Domain Test-Time Scaling
Oct 1, 2025Dong Bok Lee, Seanie Lee, Sangwoo Park +12Process Reward Model LLM Reasoning Strategies
KAIST · TH Nürnberg · Microsoft Research Asia

May 17, 2026, cs.LGJ/K move · Enter open · S save

Step-wise Rubric Rewards for LLM Reasoning

Weichu Xie, Haozhe Zhao, Wenpu Liu, Yongfu Zhu+14

Peking University · JD Explore Academy · Shanghai Jiao Tong University+2

Reinforcement Learning With Verifiable Reward Rubric-Based Scoring Faithful Explanation

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models, but rewards only final-answer correctness with no supervision over intermediate steps. Rubric-based methods such as Rubrics as Rewards (RaR) introduce finer-grained supervision by scoring rollouts against structured criteria, yet the rubric scores are still aggregated into a single scalar applied to the entire response, causing three weaknesses: loss of multi-criterion structure, uniform supervision of correct and incorrect steps, and reward hacking through unbounded self-correction. On 1,000 problems, we find 18.2% of steps in correct-answer responses are wrong yet positively rewarded, while 49.9% of steps in incorrect-answer responses are correct yet penalized. We introduce Step-wise Rubrics as Rewards (SRaR), an RLVR framework that (i) uses an LLM judge to attribute each rubric item to a specific reasoning step, (ii) normalizes per-step rubric scores across rollouts so only steps whose quality varies produce a learning signal, and (iii) combines the per-step reward with the outcome reward through a decoupled advantage estimator that keeps the outcome baseline stable. We further build a 16K-problem rubric dataset by contrastively distilling rubric items from correct and flawed reasoning paths sampled from a strong model. Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points on Qwen3-8B and 2.75 points on Qwen3-32B, raises the Faithful Reasoning Rate on AIME 2025 from 34.5% to 46.7%, and reduces self-correction looping from 48.1% to 26.5%.

Abstract

Explore similar work

Step-wise Rubric Rewards for LLM Reasoning

ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth

Rethinking Reward Models for Multi-Domain Test-Time Scaling