cs.LGOct 16, 2026

Process Reward Modeling with Reliability- and Conservatism-Aware Bayesian Belief

Authors: Zhongjian Qiao, Zhixiang Zhou, Jiafei Lyu, Peisong Wang, Qiaosheng Zhang, Siyang Gao, Shuang Qiu

Organizations: CityUHK · Shanghai Innovation Institute · Tencent · Institute of Automation, Chinese Academy of Sciences

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Process Reward Models (PRMs) play a central role in test-time scaling for large language model (LLM) reasoning. However, PRMs may provide inaccurate process rewards for test-time scaling, which we identify as arising from two sources of uncertainty: (1) PRM uncertainty that arises from insufficient supervision data and imperfect PRM training; and (2) policy uncertainty that results from the mismatch between the policy to collect supervision data and the policy for test-time scaling, leading to systematic reward overestimation. Previous studies fail to jointly consider both uncertainties. To bridge this gap, we propose BayesianPRM, which treats the process reward as a random variable within a reward hypothesis space and maintains a Bayesian belief over it. Since exact Bayesian inference over the continuous hypothesis space is generally intractable, we discretize the reward hypothesis space with a reward ensemble and introduce two complementary belief updates, each targeting one source of uncertainty: (1) to address PRM uncertainty, we introduce reliability-aware belief updating that measures how well each ensemble member explains the supervision data and updates the belief with an amortized belief network optimized by the evidence lower bound (ELBO); and (2) to tackle policy uncertainty, we propose conservatism-aware belief calibration which further reweights the belief toward lower rewards to suppress overestimation. These two updates yield a reliability- and conservatism-aware Bayesian belief, whose expectation determines the process reward. Experiments on multimodal reasoning benchmarks indicate that BayesianPRM consistently yields more accurate process reward estimates while enhancing test-time scaling performance.

Figures & tables

 PRM UncertaintyPolicy UncertaintyTechnique
BetaPRM ( Li et al., 2026b )✓✗Beta Distribution
QR-PRM ( Park et al., 2026 )✗✓Quantile Regression
OT-PRM ( Ma et al., 2026 )✗✓Optimal Transport
BayesianPRM (Ours)✓✓Bayesian Learning
Table 1: Comparison between BayesianPRM and relevant recent works.
Figure 1: Illustration of PRM uncertainty and policy uncertainty. (a)-(b) Prediction error distributions on MathV360K (ID) and Geometry3K (OOD), reflecting PRM uncertainty. (c)-(d) Prediction error distribution on UniGeo (ID) and MathVision (OOD), illustrating policy uncertainty. Prediction error is defined as the predicted reward minus the empirical success probability.
 MathVisionMathVerse-VOMathVistaMMStarAverage
Brier Score ↓\downarrow
Standard PRM0.2025 ±0.0194\pm 0.01940.2274 ±0.0143\pm 0.01430.1704 ±0.0082\pm 0.00820.2357 ±0.0236\pm 0.02360.2090
BetaPRM0.2377 ±0.0221\pm 0.02210.2489 ±0.0163\pm 0.01630.1712 ±0.0148\pm 0.01480.2427 ±0.0121\pm 0.01210.2251
DreamPRM0.2461 ±0.0292\pm 0.02920.2498 ±0.0180\pm 0.01800.1904 ±0.0064\pm 0.00640.2342 ±0.0208\pm 0.02080.2301
BayesianPRM0.1411 ±0.0201\pm 0.02010.1848 ±0.0112\pm 0.01120.1579 ±0.0107\pm 0.01070.2085 ±0.0153\pm 0.01530.1731
Positive Brier Score ↓\downarrow
Table 2: PRM accuracy evaluation results. We evaluate Brier Score, Positive Brier Score, ECE, and AverageCE on four benchmarks: MathVision, the Vision Only subset of MathVerse (denoted as MathVerse-VO), MathVista, and MMStar. ↓\downarrow indicates that lower values are better. ±\pm denotes the standard deviation across three random training seeds.
 MathVisionMathVerse-VOMathVistaMMStarAverage
InternVL3-8B29.9345.0565.9066.2751.79
Best-of-N ( N=16N=16 ) ↑\uparrow
Self-Consistency33.2247.4668.0068.6054.32
Standard PRM32.57 ±0.78\pm 0.7850.78 ±1.17\pm 1.1769.00 ±1.10\pm 1.1067.47 ±0.61\pm 0.6154.96
BetaPRM34.31 ±0.70\pm 0.7051.85 ±0.63\pm 0.6368.66 ±1.15\pm 1.1567.77 ±1.54\pm 1.5455.65
DreamPRM34.79 ±1.01\pm 1.0151.22 ±0.64\pm 0.6469.13 ±0.65\pm 0.6567.18 ±1.12\pm 1.1255.58
Table 3: PRM-guided test-time scaling performance comparison. We compare Best-of- NN ( N=16N=16 ) and Instance-Adaptive Scaling (IAS, Nmax⁡=16N_{\max}=16 ) on MathVision, MathVerse-VO, MathVista and MMStar. ↑\uparrow indicates that higher values are better. ±\pm denotes the standard deviation across three random training seeds.
Figure 5
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

ItemValue
BackboneInternVL3-8B
OptimizerAdamW ( Loshchilov and Hutter, 2017 )
Learning Rate1×10−51\times 10^{-5}
Weight Decay0.05
Learning Rate SchedulerCosine
Warmup Ratio0.05
Appendix
Table 6: Hyperparameters for BayesianPRM.
 MathVisionMathVerse-VOMathVistaMMStarAverage
Brier Score
Standard PRM0.2540 ±0.0128\pm 0.01280.2695 ±0.0210\pm 0.02100.2223 ±0.0143\pm 0.01430.2506 ±0.0088\pm 0.00880.2491
BetaPRM0.2854 ±0.0259\pm 0.02590.2901 ±0.0220\pm 0.02200.2267 ±0.0166\pm 0.01660.2540 ±0.0192\pm 0.01920.2641
DreamPRM0.2869 ±0.0234\pm 0.02340.3221 ±0.0202\pm 0.02020.2363 ±0.0051\pm 0.00510.2588 ±0.0120\pm 0.01200.2760
BayesianPRM0.2002 ±0.0187\pm 0.01870.2439 ±0.0261\pm 0.02610.2118 ±0.0242\pm 0.02420.2262 ±0.0105\pm 0.01050.2205
Positive Brier Score
Appendix
Table 7: PRM accuracy evaluation results for Qwen3VL-8B. We evaluate Brier Score, Positive Brier Score, ECE, and AverageCE on four benchmarks: MathVision, the Vision Only subset of MathVerse (denoted as MathVerse-VO), MathVista, and MMStar. ↓\downarrow indicates that lower values are better. ±\pm denotes the standard deviation across three random training seeds.
 MathVisionMathVerse-VOMathVistaMMStarAverage
Qwen3VL-8B20.0750.5170.3056.6749.39
Best-of- NN ( N=16N=16 )
Self-Consistency22.0454.1973.2059.3352.19
Standard PRM20.86 ±0.36\pm 0.3653.52 ±0.64\pm 0.6474.97 ±1.04\pm 1.0459.61 ±0.27\pm 0.2752.24
BetaPRM21.14 ±0.57\pm 0.5753.39 ±0.33\pm 0.3374.79 ±0.70\pm 0.7060.07 ±0.44\pm 0.4452.35
DreamPRM21.06 ±0.62\pm 0.6253.71 ±0.20\pm 0.2074.15 ±1.22\pm 1.2260.27 ±0.65\pm 0.6552.30
Appendix
Table 8: PRM-guided test-time scaling performance comparison. We compare Best-of- NN ( N=16N=16 ) and Instance-Adaptive Scaling (IAS, Nmax⁡=16N_{\max}=16 ) on MathVision, MathVerse-VO, MathVista and MMStar. ↑\uparrow indicates that higher values are better. ±\pm denotes the standard deviation across three random training seeds.
ModelOverallMathVisionMathVerseMMMUDynaMathWeMath
Standard PRM59.79 ±0.41\pm 0.4157.03 ±0.16\pm 0.1659.65 ±0.26\pm 0.2662.23 ±0.42\pm 0.4259.71 ±0.24\pm 0.2463.13 ±0.60\pm 0.60
BetaPRM60.83 ±0.23\pm 0.2357.83 ±0.38\pm 0.3860.40 ±0.44\pm 0.4460.59 ±0.26\pm 0.2661.33 ±0.62\pm 0.6263.47 ±0.48\pm 0.48
DreamPRM60.69 ±0.32\pm 0.3258.13 ±0.47\pm 0.4759.44 ±0.29\pm 0.2961.22 ±0.54\pm 0.5461.06 ±0.15\pm 0.1563.59 ±0.27\pm 0.27
BayesianPRM62.23 ±0.29\pm 0.2961.58 ±0.54\pm 0.5461.31 ±0.31\pm 0.3163.39 ±0.36\pm 0.3661.12 ±0.47\pm 0.4764.06 ±0.22\pm 0.22
Appendix
Table 9: Evaluation results on VisualProcessBench. Each source column reports macro-F1 on the corresponding subset, while Overall denotes the micro-F1 computed over all non-neutral annotated steps. ±\pm denotes the standard deviation across three random training seeds.
 MathVisionMathVerse-VOMathVistaMMStarAverage
Brier Score ↓\downarrow
QR-PRM0.1634 ±0.0166\pm 0.01660.1899 ±0.0147\pm 0.01470.1343 ±0.0124\pm 0.01240.1795 ±0.0262\pm 0.02620.1668
TS0.2931 ±0.0216\pm 0.02160.2652 ±0.0124\pm 0.01240.2145 ±0.0189\pm 0.01890.2667 ±0.0160\pm 0.01600.2599
IR0.2568 ±0.0357\pm 0.03570.2420 ±0.0324\pm 0.03240.2001 ±0.0164\pm 0.01640.2504 ±0.0293\pm 0.02930.2373
HB0.3146 ±0.0250\pm 0.02500.2847 ±0.0127\pm 0.01270.2394 ±0.0165\pm 0.01650.2781 ±0.0277\pm 0.02770.2792
BayesianPRM0.1035 ±0.0142\pm 0.01420.1422 ±0.0169\pm 0.01690.0946 ±0.0125\pm 0.01250.1521 ±0.0178\pm 0.01780.1231
Appendix
Table 10: PRM accuracy evaluation results with calibration sets. We evaluate Brier Score, Positive Brier Score, ECE, and AverageCE on four benchmarks: MathVision, the Vision Only subset of MathVerse (denoted as MathVerse-VO), MathVista, and MMStar. TS, IR, HB denote Temperature Scaling, Isotonic Regression, Histogram Binning, respectively. ↓\downarrow indicates that lower values are better. ±\pm denotes the standard deviation across three random training seeds.
 MathVisionMathVerse-VOMathVistaMMStarAverage
Best-of- NN ( N=16N=16 ) ↑\uparrow
QR-PRM34.79 ±0.67\pm 0.6751.26 ±0.84\pm 0.8469.28 ±0.73\pm 0.7368.71 ±0.40\pm 0.4056.01
TS32.10 ±1.18\pm 1.1849.66 ±0.64\pm 0.6469.12 ±0.48\pm 0.4867.01 ±0.39\pm 0.3954.47
IR32.67 ±1.14\pm 1.1451.26 ±0.79\pm 0.7968.83 ±0.85\pm 0.8567.58 ±0.30\pm 0.3055.09
HB32.00 ±1.01\pm 1.0149.13 ±0.57\pm 0.5769.06 ±0.82\pm 0.8267.24 ±1.13\pm 1.1354.36
BayesianPRM35.80 ±0.56\pm 0.5651.93 ±0.77\pm 0.7769.56 ±0.33\pm 0.3369.69 ±0.90\pm 0.9056.75
Appendix
Table 11: PRM-guided test-time scaling performance comparison with calibration sets. We compare Best-of- NN ( N=16N=16 ) and Instance-Adaptive Scaling (IAS, Nmax⁡=16N_{\max}=16 ) on MathVision, MathVerse-VO, MathVista and MMStar. ↑\uparrow indicates that higher values are better. ±\pm denotes the standard deviation across three random training seeds.
 MathVisionMathVerse-VOMathVistaMMStarAverage
Vanilla Ensemble PRM34.88 ±0.47\pm 0.4750.21 ±0.92\pm 0.9268.26 ±0.59\pm 0.5968.03 ±0.88\pm 0.8855.35
BayesianPRM35.86 ±0.70\pm 0.7051.40 ±1.11\pm 1.1169.40 ±0.53\pm 0.5369.33 ±0.60\pm 0.6056.50
Appendix
Table 12: Comparison with Vanilla Ensemble PRM . We compare Best-of- NN ( N=16N=16 ) on MathVision, MathVerse-VO, MathVista and MMStar. ±\pm denotes the standard deviation across three random training seeds.
Figure 4: Reliability-aware belief distribution during training. We use M=5M=5 for visualization clarity.
 MathVisionMathVerse-VOMathVistaMMStarAverage
β1=0.001\beta_{1}=0.00134.28 ±0.46\pm 0.4650.30 ±0.88\pm 0.8868.33 ±0.67\pm 0.6768.56 ±0.83\pm 0.8355.37
β1=0.01\beta_{1}=0.0135.67 ±0.78\pm 0.7851.86 ±1.27\pm 1.2769.32 ±0.42\pm 0.4269.07 ±0.57\pm 0.5756.48
β1=0.05\beta_{1}=0.0535.86 ±0.70\pm 0.7051.40 ±1.11\pm 1.1169.40 ±0.53\pm 0.5369.33 ±0.60\pm 0.6056.50
β1=0.1\beta_{1}=0.135.21 ±0.40\pm 0.4050.35 ±0.63\pm 0.6368.82 ±0.27\pm 0.2768.19 ±0.64\pm 0.6455.64
Appendix
Table 13: Ablation Study on β1\beta_{1} . We compare Best-of- NN ( N=16N=16 ) on MathVision, MathVerse-VO, MathVista and MMStar. ±\pm denotes the standard deviation across three random training seeds.
 MathVisionMathVerse-VOMathVistaMMStarAverage
β2=0.001\beta_{2}=0.00135.24 ±0.62\pm 0.6250.62 ±0.94\pm 0.9468.43 ±0.46\pm 0.4669.10 ±0.82\pm 0.8255.85
β2=0.01\beta_{2}=0.0135.78 ±0.76\pm 0.7650.54 ±0.85\pm 0.8569.21 ±0.50\pm 0.5069.39 ±0.74\pm 0.7456.23
β2=0.05\beta_{2}=0.0535.86 ±0.70\pm 0.7051.40 ±1.11\pm 1.1169.40 ±0.53\pm 0.5369.33 ±0.60\pm 0.6056.50
β2=0.1\beta_{2}=0.134.72 ±0.49\pm 0.4950.26 ±1.26\pm 1.2669.27 ±0.61\pm 0.6169.98 ±0.85\pm 0.8556.06
Appendix
Table 14: Ablation Study on β2\beta_{2} . We compare Best-of- NN ( N=16N=16 ) on MathVision, MathVerse-VO, MathVista and MMStar. ±\pm denotes the standard deviation across three random training seeds.
 MathVisionMathVerse-VOMathVistaMMStarAverage
M=10M=1035.30 ±0.72\pm 0.7251.42 ±0.79\pm 0.7968.97 ±0.42\pm 0.4268.49 ±0.59\pm 0.5956.05
M=20M=2035.86 ±0.70\pm 0.7051.40 ±1.11\pm 1.1169.40 ±0.53\pm 0.5369.33 ±0.60\pm 0.6056.50
M=50M=5035.89 ±0.65\pm 0.6551.35 ±1.30\pm 1.3069.52 ±0.49\pm 0.4968.99 ±0.64\pm 0.6456.44
M=100M=10035.87 ±0.58\pm 0.5851.46 ±0.83\pm 0.8369.34 ±0.60\pm 0.6069.62 ±0.39\pm 0.3956.57
Appendix
Table 15: Ablation Study on MM . We compare Best-of- NN ( N=16N=16 ) on MathVision, MathVerse-VO, MathVista and MMStar. ±\pm denotes the standard deviation across three random training seeds.
 MathVisionMathVerse-VOMathVistaMMStarAverage
λprior=0.1\lambda_{\text{prior}}=0.134.93 ±0.49\pm 0.4949.84 ±0.53\pm 0.5368.20 ±0.73\pm 0.7367.77 ±0.43\pm 0.4355.19
λprior=1\lambda_{\text{prior}}=135.39 ±0.79\pm 0.7950.27 ±0.88\pm 0.8868.85 ±0.56\pm 0.5669.12 ±0.93\pm 0.9355.91
λprior=10\lambda_{\text{prior}}=1035.86 ±0.70\pm 0.7051.40 ±1.11\pm 1.1169.40 ±0.53\pm 0.5369.33 ±0.60\pm 0.6056.50
λprior=50\lambda_{\text{prior}}=5035.41 ±1.09\pm 1.0951.02 ±0.69\pm 0.6968.82 ±0.64\pm 0.6469.24 ±0.48\pm 0.4856.12
Appendix
Table 16: Ablation Study on λprior\lambda_{\text{prior}} . We compare Best-of- NN ( N=16N=16 ) on MathVision, MathVerse-VO, MathVista and MMStar. ±\pm denotes the standard deviation across three random training seeds.
Figure 5: A case study of BayesianPRM’s step-wise evaluation.
More in this topic

MLLM Reasoning

See all 24 papers

Explore similar work

CardsList
May 11, 2026

Unsupervised Process Reward Models

Process Reward Models (PRMs) are a powerful mechanism for steering large language model reasoning by providing fine-grained, step-level supervision. However, this effectiveness comes at a significant cost: PRMs require expert annotations for every reasoning step, making them costly and difficult to scale. Here, we propose a method for training unsupervised PRMs (uPRM) that requires no human supervision, neither at the level of step-by-step annotations nor through ground-truth verification of final answers. The key idea behind our approach is to define a scoring function, derived from LLM next-token probabilities, that jointly assesses candidate positions of first erroneous steps across a batch of reasoning trajectories. We demonstrate the effectiveness of uPRM across diverse scenarios: (i) uPRM achieves up to 15% absolute accuracy improvements over the LLM-as-a-Judge in identifying first erroneous steps on the ProcessBench dataset; (ii) as a verifier for test-time scaling, uPRM performs comparably to supervised PRMs and outperforms the majority voting baseline by up to 6.9%, and (iii) when used as a reward signal in reinforcement learning, uPRM enables more robust policy optimization throughout training compared to a supervised PRM trained using ground-truth labels. Overall, our results open a path toward scalable reward modeling for complex reasoning tasks.

Artyom Gadetsky, Maxim Kodryan, Siba Smarak Panigrahi +2

Swiss Federal Institute of Technology (EPFL)

Reward ModelingProcess Reward ModelsReasoning Verification
AskPDF
Dec 2, 2025

ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth

Training process reward models (PRMs) requires step-level correctness labels, obtained either through expensive human annotation or by relying on ground-truth answers, limiting the ability to scale process-level supervision. We propose ScalePRM, which scales verification compute as an alternative: given a problem and a candidate solution, we generate multiple independent verifications of each reasoning step and aggregate their judgments to produce synthetic step-level labels without ground truth. We explore two representative inference-time scaling strategies, parallel scaling through self-consistency and sequential scaling through meta-critique, and train generative PRMs on the resulting synthetic data. On ProcessBench, a benchmark for identifying erroneous steps in mathematical reasoning, PRMs trained on step-level self-consistency data achieve 67.5 F1, surpassing reference-guided training with ground-truth access (66.4 F1) and GPT-4o as a critic (61.9 F1). When deployed as reward signals in RL training with Qwen2.5-Math-7B, our best PRM achieves 47.4% average accuracy across six mathematical reasoning benchmarks, outperforming ground-truth-based RLVR (43.9%). We also identify and address reward exploitation patterns unique to generative PRM-based RL. Our results demonstrate that scaling verification compute is a viable alternative to ground-truth supervision for training process reward models.

Salman Rahman, Sruthi Gorantla, Arpit Gupta +3

2UCLA · Work done while as an intern at Amazon AGI. · 1Amazon AGI

Process Reward ModelsReinforcement Learning with Verifiable RewardsLLM Mathematical Reasoning
AskPDF
Jun 8, 2026

The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning

Process Reward Models (PRMs) improve credit assignment for reasoning by providing step-level feedback. However, we identify a hidden bias in PRMs caused by severe imbalance in step-level training data. Standard cross-entropy training amplifies this bias, causing PRMs to overcredit plausible but incorrect steps and produce high false-positive rates. We show that these false positives have an asymmetric downstream effect: false negatives mainly slow exploration, whereas false positives actively steer Best-of-N selection, guided decoding, and policy optimization toward flawed reasoning. This suggests that PRM training should shift from pointwise label fitting to reliable relative comparisons. To address this, we propose PRISM (Precision Ranking for Improved Step Modeling), a policy-aware PRM training framework that learns from contrastive step-level comparisons and hard negatives generated by a temporal lookahead strategy, requiring no new human labels. We further use a difficulty-aware curriculum to optimize the contrastive step margin. Across PRMBench and ProcessBench, PRISM substantially reduces false positives (22% on PRMBench) and improves macro F1 over strong discriminative PRMs. When applied to policy optimization and search tasks, including guided decoding and Best-of-N selection, it consistently improves accuracy (up to 22% for guided decoding and 33% for Best-of-N) and robustness. More broadly, trustworthy process supervision is not just about assigning high rewards, but about rewarding the right reasoning for the right reasons.

Aakriti Agrawal, Souradip Chakraborty, Armin Saghafian +6

University of Maryland, College Park · Capital One · University of Central Florida

Process Reward ModelsRL for Language Model ReasoningStep-Level Credit Assignment in RLStep-Level Credit Assignment
AskPDF