Organizations: School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, Guangdong, China · Shenzhen Loop Area Institute, Shenzhen, Guangdong, China · Meta AI, Menlo Park, California, USA · Computer Science Department, University of California, Davis, Davis, California, USA
Process reward models (PRMs) provide fine-grained supervision for reasoning, but reliable PRMs often require step annotations or heavy verification pipelines, making them costly to scale and refresh during online RL. Implicit PRMs reduce this cost by training log-likelihood-ratio rewards from trajectory-level outcome labels. However, the log-ratio is constrained only as a sequence-level aggregate during training, while inference decomposes it into token- or step-level scores for partial prefixes. This train-inference mismatch leaves local credits weakly identified, so distribution-wide scoring can amplify misleading advantages. We propose Implicit Prefix-Value Reward Model (IPVRM), which directly learns the probability of eventual correctness for each prefix from outcome labels. Step signals are then obtained as temporal-difference (TD) differences between consecutive prefix values, aligning the training target with inference-time use. IPVRM markedly improves step-verification F1 on ProcessBench. To exploit these prefix values during policy optimization, we further introduce Distribution-Level RL (DistRL), which applies TD advantages to both sampled tokens and high-probability candidate tokens, providing dense counterfactual updates without additional rollouts. Experiments show that DistRL brings limited gains with unreliable implicit rewards, but consistently improves downstream reasoning when paired with IPVRM. The implementation of our method is available at https://github.com/gaoshiping/IPVRM .
Figures & tables
Figure 1: Paradigm comparison and qualitative trade-offs. Existing paradigms either incur high annotation/inference costs (Explicit/Generative PRMs) or suffer from a train–inference objective mismatch (Implicit PRMs). Our proposed IPVRM resolves these limitations by directly learning prefix-conditioned values Vϕ(st) , achieving both low-cost efficiency and training consistency (”what we train is what we use”) to support Distribution-Level RL.
Figure 2: Performance comparison on Qwen3-0.6B. IPVRM improves both step-level error localization on ProcessBench (Left) and sequence-level reranking in Best-of- N selection (Middle) over prior implicit reward models. Leveraging these better signals, DistRL further outperforms standard GRPO in downstream policy performance (AVG@8) (Right).
Figure 3: Overview of the IPVRM+DistRL framework.
Figure 4: Illustration of DistRL. Unlike standard RL that optimizes solely on the sampled trajectory (red bar), DistRL introduces a dual-branch mechanism. It simultaneously updates high-probability candidate tokens (blue bars) using dense one-step TD advantages, while refining the sampled token (red bar) using a hybrid of GAE and TD advantages derived from IPVRM estimates.
Figure 5: Schematic of Adaptive Difficulty Boundary (ADB) and Dynamic Loss Weighting (DLW). As illustrated, ADB shifts the sigmoid decision boundary by V(x) (the expected accuracy), while DLW reweights samples. This focuses learning on hard negatives for easy prompts (left) and hard positives for hard prompts (right), mitigating label imbalance.
Base Model
RM Method
MATH-500
Minerva-Math
Avg.
@4
@16
@64
@4
@16
@64
Qwen3-0.6B Base
Q-RM
41.4
44.4
44.6
9.2
12.1
10.7
27.1
EndoRM
32.0
33.8
38.8
6.3
8.8
7.7
21.2
DPO-RM
41.6
42.6
42.2
10.7
9.9
11.0
26.3
Implicit PRM
39.6
39.6
38.6
10.3
11.4
11.0
25.1
IPVRM
42.0
46.0
47.6
10.3
11.0
11.4
28.1
Table 1: Different reward models’ best-of-N sampling performance on MATH-500 and Minerva-Math with SFT models (Avg is computed by averaging over the two datasets, where each dataset score is the mean over @4/@16/@64).
Base Model
RM Method
GSM8K
MATH
Olympiad Bench
Omni Math
Avg.
Qwen3-0.6B base
Q-RM
47.2
38.9
31.7
35.0
38.2
EndoRM
33.7
30.1
26.9
31.5
30.6
DPO-RM
32.1
32.7
23.5
22.9
27.8
Implicit PRM
30.3
31.0
21.5
18.8
25.4
IPVRM
52.8
42.1
30.0
31.3
39.1
Qwen3-8B base
Q-RM
61.0
45.2
39.5
35.3
45.3
Table 2: Performance ( F1 score) of different reward modeling methods on ProcessBench .
Base Model
RM Method
MATH-500
Avg.
@4
@16
@64
Qwen3-0.6B base
IPVRM late
42.4
47.0
50.0
46.5
IPVRM
42.0
46.0
47.6
45.2
IPVRM early
41.0
44.2
44.2
43.1
Llama3.2-1B base
IPVRM late
14.4
14.6
17.0
15.3
IPVRM
13.9
14.2
14.8
14.3
Table 3: Variants of IPVRM: best-of-N sampling performance on MATH-500 with SFT model.
Base Model
RM Method
GSM8K
MATH
Olympiad Bench
Omni Math
Avg.
Qwen3-0.6B base
IPVRM late
50.8
40.3
28.4
29.2
37.2
IPVRM
52.8
42.1
30.0
31.3
39.1
IPVRM early
49.9
43.4
32.2
34.8
40.1
Llama3.2-1B base
IPVRM late
41.1
33.8
27.1
27.6
32.4
IPVRM
42.7
35.2
28.7
29.2
34.0
IPVRM early
45.0
34.8
30.1
31.4
35.3
Table 4: Performance ( F1 score) of different IPVRM variants on ProcessBench .
Base Model
Method
AIME
MATH 500
Minerva Math
Olympiad Bench
AMC
Avg.
Qwen3-0.6B base
SFT
0.4
33.2
7.3
7.6
9.3
11.6
PRIME
1.1
39.6
11.4
11.4
12.3
15.2
SPRO
1.7
38.6
10.7
11.6
13.7
15.3
Reinforce w/ Q-RM
1.1
36.4
9.2
11.0
10.8
13.7
GRPO w/ VR
0.8
38.3
10.5
10.5
12.5
14.5
GRPO w/ DPO-RM
0.8
38.4
9.8
10.0
11.1
14.0
Table 5: Main results for policy model (AVG@8)
Base Model
Method
AIME
MATH 500
Minerva Math
Olympiad Bench
AMC
Avg.
Qwen3-0.6B base
GRPO w/ DPO-RM
0.8
38.4
9.8
10.0
11.1
14.0
DistRL w/ DPO-RM
0.8
37.2
9.4
10.1
13.0
14.1
GRPO w/ Implicit PRM
1.2
34.9
10.2
9.5
10.8
13.3
DistRL w/ Implicit PRM
0.8
37.1
10.9
10.5
14.2
14.7
GRPO w/ IPVRM
1.2
41.5
12.1
12.9
13.9
16.3
DistRL w/ IPVRM
1.5
42.5
13.3
12.1
14.0
16.7
Table 6: Impact of reward models and DistRL across mathematical benchmarks (AVG@8).
RM score
Verifier acc.
Qwen3-0.6B
−0.140 / −0.181
23.5 / 23.0
Qwen3-8B
−0.178 / −0.202
41.6 / 41.4
Llama3.2-1B
−0.040 / −0.050
7.7 / 7.4
Llama3.2-3B
−0.024 / −0.025
15.9 / 15.5
Table 7: Training-time analysis of TD. DistRL achieves higher RM score and verifier accuracy than GRPO across all backbones. Each entry is reported as DistRL / GRPO .
Figure 6: Ablation of online IPVRM update strategies on Llama3.2-1B. Left: outcome training reward during RL. Right: RM accuracy across training samples ( ×16384 ).
Method
AIME
MATH500
Minerva
Olympiad
AMC
Avg
Implicit PRM w/o ADB/DLW
0.8
37.1
10.9
10.5
14.2
14.7
IPVRM w/o ADB/DLW
1.0
40.8
11.2
12.2
12.0
15.4
IPVRM w/ ADB/DLW
1.5
42.5
13.3
12.1
14.0
16.7
Table 8: Policy performance under different online RM update settings on Qwen3-0.6B with DistRL.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Base Model
Reference Choice
AIME
MATH 500
Minerva Math
Olympiad Bench
AMC
Avg.
Qwen3-0.6B base
SFT Model
1.4
39.8
12.1
11.3
11.3
15.2
Behavior Policy
1.5
42.5
13.3
12.1
14.0
16.7
Llama3.2-1B base
SFT Model
0.3
10.3
2.9
3.5
2.1
3.8
Behavior Policy
0.1
13.2
3.4
3.1
4.2
4.8
Appendix
Table 9: Impact of reference choice on distribution-level RL across mathematical benchmarks (AVG@8).
m
BoN-AVG
PB-AVG
0
26.1
37.0
5
28.1
39.1
10
27.5
39.1
Appendix
Table 10: Sensitivity to the margin m on Qwen3-0.6B.
Pmin
Avg Candidate Size
Prob. Mass Coverage
0.05
1.54
97.57%
0.10
1.33
96.11%
0.20
1.16
93.58%
Appendix
Table 11: Sensitivity to the candidate threshold Pmin on Qwen3-0.6B.
Table 12: Best-of- N sampling accuracy (%) on AMC-23 with SFT models.
Model
Method
Peak Memory (GB)
Avg Time / Step (s)
Qwen3-0.6B
GRPO
30.20
487.5 ± 27.1
DistRL
30.88
458.7 ± 33.2
Llama3.2-1B
GRPO
31.30
349.8 ± 12.6
DistRL
31.52
319.6 ± 13.0
Appendix
Table 13: Peak GPU memory usage and average wall-clock time per training step for IPVRM with DistRL and GRPO.
Protocol
Method
GSM8K
MATH
Olympiad
OmniMath
Avg
Prefix-based
DPO-RM
27.0
27.4
19.8
18.4
23.2
Implicit PRM
30.3
28.2
19.9
16.9
23.8
IPVRM (Ours)
42.6
36.4
24.3
25.2
32.1
Process-level
DPO-RM
32.1
32.7
23.5
22.9
27.8
Implicit PRM
30.3
31.0
21.5
18.8
25.4
IPVRM (Ours)
52.8
42.1
30.0
31.3
39.1
Appendix
Table 14: ProcessBench F1 under different evaluation protocols.
Process reward models (PRMs) have demonstrated notable effectiveness in test-time scaling and reinforcement learning by providing fine-grained signals for evaluating intermediate reasoning states, but their training relies heavily on costly process annotations. A natural way to alleviate this dependence is to complement limited process supervision with scalable outcome supervision. However, existing PRMs often model reasoning prefixes independently, providing no explicit mechanism for effectively using final outcome to guide the learning of intermediate reasoning states. We introduce Reasoning State Propagation (RSP), which represents each reasoning prefix with a binary validity state and models transitions between successive states across the reasoning trajectory. Specifically, RSP predicts a break probability that a valid state becomes invalid and a repair probability that an invalid state returns to valid. By propagating these transitions, RSP connects intermediate states to the final state, allowing process annotations to supervise intermediate states while outcome labels supervise the final state and can provide learning signals to preceding steps. Across reasoning search, response selection, and reinforcement learning, RSP consistently outperforms representative PRM baselines, with average improvements over Qwen2.5-Math-PRM of 5.6% in beam search and 2.1% in reinforcement learning.
Kai Gan, Zi-Hao Zhou, Bo Ye +3
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China · Zhongguancun Academy +1
Process Reward Models (PRMs) are a powerful mechanism for steering large language model reasoning by providing fine-grained, step-level supervision. However, this effectiveness comes at a significant cost: PRMs require expert annotations for every reasoning step, making them costly and difficult to scale. Here, we propose a method for training unsupervised PRMs (uPRM) that requires no human supervision, neither at the level of step-by-step annotations nor through ground-truth verification of final answers. The key idea behind our approach is to define a scoring function, derived from LLM next-token probabilities, that jointly assesses candidate positions of first erroneous steps across a batch of reasoning trajectories. We demonstrate the effectiveness of uPRM across diverse scenarios: (i) uPRM achieves up to 15% absolute accuracy improvements over the LLM-as-a-Judge in identifying first erroneous steps on the ProcessBench dataset; (ii) as a verifier for test-time scaling, uPRM performs comparably to supervised PRMs and outperforms the majority voting baseline by up to 6.9%, and (iii) when used as a reward signal in reinforcement learning, uPRM enables more robust policy optimization throughout training compared to a supervised PRM trained using ground-truth labels. Overall, our results open a path toward scalable reward modeling for complex reasoning tasks.
Process Reward Models (PRMs) improve credit assignment for reasoning by providing step-level feedback. However, we identify a hidden bias in PRMs caused by severe imbalance in step-level training data. Standard cross-entropy training amplifies this bias, causing PRMs to overcredit plausible but incorrect steps and produce high false-positive rates. We show that these false positives have an asymmetric downstream effect: false negatives mainly slow exploration, whereas false positives actively steer Best-of-N selection, guided decoding, and policy optimization toward flawed reasoning. This suggests that PRM training should shift from pointwise label fitting to reliable relative comparisons. To address this, we propose PRISM (Precision Ranking for Improved Step Modeling), a policy-aware PRM training framework that learns from contrastive step-level comparisons and hard negatives generated by a temporal lookahead strategy, requiring no new human labels. We further use a difficulty-aware curriculum to optimize the contrastive step margin. Across PRMBench and ProcessBench, PRISM substantially reduces false positives (22% on PRMBench) and improves macro F1 over strong discriminative PRMs. When applied to policy optimization and search tasks, including guided decoding and Best-of-N selection, it consistently improves accuracy (up to 22% for guided decoding and 33% for Best-of-N) and robustness. More broadly, trustworthy process supervision is not just about assigning high rewards, but about rewarding the right reasoning for the right reasons.