Process Reward Modeling with Reliability- and Conservatism-Aware Bayesian Belief

Zhongjian Qiao; Zhixiang Zhou; Jiafei Lyu; Peisong Wang; Qiaosheng Zhang; Siyang Gao; Shuang Qiu

cs.LGOct 16, 2026

Process Reward Modeling with Reliability- and Conservatism-Aware Bayesian Belief

Authors: Zhongjian Qiao, Zhixiang Zhou, Jiafei Lyu, Peisong Wang, Qiaosheng Zhang, Siyang Gao, Shuang Qiu

Organizations: CityUHK · Shanghai Innovation Institute · Tencent · Institute of Automation, Chinese Academy of Sciences

Abstract

Process Reward Models (PRMs) play a central role in test-time scaling for large language model (LLM) reasoning. However, PRMs may provide inaccurate process rewards for test-time scaling, which we identify as arising from two sources of uncertainty: (1) PRM uncertainty that arises from insufficient supervision data and imperfect PRM training; and (2) policy uncertainty that results from the mismatch between the policy to collect supervision data and the policy for test-time scaling, leading to systematic reward overestimation. Previous studies fail to jointly consider both uncertainties. To bridge this gap, we propose BayesianPRM, which treats the process reward as a random variable within a reward hypothesis space and maintains a Bayesian belief over it. Since exact Bayesian inference over the continuous hypothesis space is generally intractable, we discretize the reward hypothesis space with a reward ensemble and introduce two complementary belief updates, each targeting one source of uncertainty: (1) to address PRM uncertainty, we introduce reliability-aware belief updating that measures how well each ensemble member explains the supervision data and updates the belief with an amortized belief network optimized by the evidence lower bound (ELBO); and (2) to tackle policy uncertainty, we propose conservatism-aware belief calibration which further reweights the belief toward lower rewards to suppress overestimation. These two updates yield a reliability- and conservatism-aware Bayesian belief, whose expectation determines the process reward. Experiments on multimodal reasoning benchmarks indicate that BayesianPRM consistently yields more accurate process reward estimates while enhancing test-time scaling performance.

Figures & tables

	PRM Uncertainty	Policy Uncertainty	Technique
BetaPRM ( Li et al., 2026b )	✓	✗	Beta Distribution
QR-PRM ( Park et al., 2026 )	✗	✓	Quantile Regression
OT-PRM ( Ma et al., 2026 )	✗	✓	Optimal Transport
BayesianPRM (Ours)	✓	✓	Bayesian Learning

Table 1: Comparison between BayesianPRM and relevant recent works.

Panel of Figure 1 — Figure 1: Illustration of PRM uncertainty and policy uncertainty. (a)-(b) Prediction error distributions on MathV360K (ID) and Geometry3K (OOD), reflecting PRM uncertainty. (c)-(d) Prediction error distribution on UniGeo (ID) and MathVision (OOD), illustrating policy uncertainty. Prediction error is defined as the predicted reward minus the empirical success probability.

Brier Score $\downarrow$
	MathVision	MathVerse-VO	MathVista	MMStar	Average
Standard PRM	0.2025 $\pm 0.0194$	0.2274 $\pm 0.0143$	0.1704 $\pm 0.0082$	0.2357 $\pm 0.0236$	0.2090
BetaPRM	0.2377 $\pm 0.0221$	0.2489 $\pm 0.0163$	0.1712 $\pm 0.0148$	0.2427 $\pm 0.0121$	0.2251
DreamPRM	0.2461 $\pm 0.0292$	0.2498 $\pm 0.0180$	0.1904 $\pm 0.0064$	0.2342 $\pm 0.0208$	0.2301
BayesianPRM	0.1411 $\pm 0.0201$	0.1848 $\pm 0.0112$	0.1579 $\pm 0.0107$	0.2085 $\pm 0.0153$	0.1731
Positive Brier Score $\downarrow$

Table 2: PRM accuracy evaluation results. We evaluate Brier Score, Positive Brier Score, ECE, and AverageCE on four benchmarks: MathVision, the Vision Only subset of MathVerse (denoted as MathVerse-VO), MathVista, and MMStar.

\downarrow

indicates that lower values are better.

\pm

denotes the standard deviation across three random training seeds.

Best-of-N ( $N=16$ ) $\uparrow$
	MathVision	MathVerse-VO	MathVista	MMStar	Average
InternVL3-8B	29.93	45.05	65.90	66.27	51.79
Self-Consistency	33.22	47.46	68.00	68.60	54.32
Standard PRM	32.57 $\pm 0.78$	50.78 $\pm 1.17$	69.00 $\pm 1.10$	67.47 $\pm 0.61$	54.96
BetaPRM	34.31 $\pm 0.70$	51.85 $\pm 0.63$	68.66 $\pm 1.15$	67.77 $\pm 1.54$	55.65
DreamPRM	34.79 $\pm 1.01$	51.22 $\pm 0.64$	69.13 $\pm 0.65$	67.18 $\pm 1.12$	55.58

Table 3: PRM-guided test-time scaling performance comparison. We compare Best-of-

N

(

N=16

) and Instance-Adaptive Scaling (IAS,

N_{\max}=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\uparrow

indicates that higher values are better.

\pm

denotes the standard deviation across three random training seeds.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
Backbone	InternVL3-8B
Optimizer	AdamW ( Loshchilov and Hutter, 2017 )
Learning Rate	$1\times 10^{-5}$
Weight Decay	0.05
Learning Rate Scheduler	Cosine
Warmup Ratio	0.05

Appendix

Table 6: Hyperparameters for BayesianPRM.

	MathVision	MathVerse-VO	MathVista	MMStar	Average
Brier Score
Standard PRM	0.2540 $\pm 0.0128$	0.2695 $\pm 0.0210$	0.2223 $\pm 0.0143$	0.2506 $\pm 0.0088$	0.2491
BetaPRM	0.2854 $\pm 0.0259$	0.2901 $\pm 0.0220$	0.2267 $\pm 0.0166$	0.2540 $\pm 0.0192$	0.2641
DreamPRM	0.2869 $\pm 0.0234$	0.3221 $\pm 0.0202$	0.2363 $\pm 0.0051$	0.2588 $\pm 0.0120$	0.2760
BayesianPRM	0.2002 $\pm 0.0187$	0.2439 $\pm 0.0261$	0.2118 $\pm 0.0242$	0.2262 $\pm 0.0105$	0.2205
Positive Brier Score

Appendix

Table 7: PRM accuracy evaluation results for Qwen3VL-8B. We evaluate Brier Score, Positive Brier Score, ECE, and AverageCE on four benchmarks: MathVision, the Vision Only subset of MathVerse (denoted as MathVerse-VO), MathVista, and MMStar.

\downarrow

indicates that lower values are better.

\pm

denotes the standard deviation across three random training seeds.

Best-of- $N$ ( $N=16$ )
	MathVision	MathVerse-VO	MathVista	MMStar	Average
Qwen3VL-8B	20.07	50.51	70.30	56.67	49.39
Self-Consistency	22.04	54.19	73.20	59.33	52.19
Standard PRM	20.86 $\pm 0.36$	53.52 $\pm 0.64$	74.97 $\pm 1.04$	59.61 $\pm 0.27$	52.24
BetaPRM	21.14 $\pm 0.57$	53.39 $\pm 0.33$	74.79 $\pm 0.70$	60.07 $\pm 0.44$	52.35
DreamPRM	21.06 $\pm 0.62$	53.71 $\pm 0.20$	74.15 $\pm 1.22$	60.27 $\pm 0.65$	52.30

Appendix

Table 8: PRM-guided test-time scaling performance comparison. We compare Best-of-

N

(

N=16

) and Instance-Adaptive Scaling (IAS,

N_{\max}=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\uparrow

indicates that higher values are better.

\pm

denotes the standard deviation across three random training seeds.

Model	Overall	MathVision	MathVerse	MMMU	DynaMath	WeMath
Standard PRM	59.79 $\pm 0.41$	57.03 $\pm 0.16$	59.65 $\pm 0.26$	62.23 $\pm 0.42$	59.71 $\pm 0.24$	63.13 $\pm 0.60$
BetaPRM	60.83 $\pm 0.23$	57.83 $\pm 0.38$	60.40 $\pm 0.44$	60.59 $\pm 0.26$	61.33 $\pm 0.62$	63.47 $\pm 0.48$
DreamPRM	60.69 $\pm 0.32$	58.13 $\pm 0.47$	59.44 $\pm 0.29$	61.22 $\pm 0.54$	61.06 $\pm 0.15$	63.59 $\pm 0.27$
BayesianPRM	62.23 $\pm 0.29$	61.58 $\pm 0.54$	61.31 $\pm 0.31$	63.39 $\pm 0.36$	61.12 $\pm 0.47$	64.06 $\pm 0.22$

Appendix

Table 9: Evaluation results on VisualProcessBench. Each source column reports macro-F1 on the corresponding subset, while Overall denotes the micro-F1 computed over all non-neutral annotated steps.

\pm

denotes the standard deviation across three random training seeds.

Brier Score $\downarrow$
	MathVision	MathVerse-VO	MathVista	MMStar	Average
QR-PRM	0.1634 $\pm 0.0166$	0.1899 $\pm 0.0147$	0.1343 $\pm 0.0124$	0.1795 $\pm 0.0262$	0.1668
TS	0.2931 $\pm 0.0216$	0.2652 $\pm 0.0124$	0.2145 $\pm 0.0189$	0.2667 $\pm 0.0160$	0.2599
IR	0.2568 $\pm 0.0357$	0.2420 $\pm 0.0324$	0.2001 $\pm 0.0164$	0.2504 $\pm 0.0293$	0.2373
HB	0.3146 $\pm 0.0250$	0.2847 $\pm 0.0127$	0.2394 $\pm 0.0165$	0.2781 $\pm 0.0277$	0.2792
BayesianPRM	0.1035 $\pm 0.0142$	0.1422 $\pm 0.0169$	0.0946 $\pm 0.0125$	0.1521 $\pm 0.0178$	0.1231

Appendix

Table 10: PRM accuracy evaluation results with calibration sets. We evaluate Brier Score, Positive Brier Score, ECE, and AverageCE on four benchmarks: MathVision, the Vision Only subset of MathVerse (denoted as MathVerse-VO), MathVista, and MMStar. TS, IR, HB denote Temperature Scaling, Isotonic Regression, Histogram Binning, respectively.

\downarrow

indicates that lower values are better.

\pm

denotes the standard deviation across three random training seeds.

Best-of- $N$ ( $N=16$ ) $\uparrow$
	MathVision	MathVerse-VO	MathVista	MMStar	Average
QR-PRM	34.79 $\pm 0.67$	51.26 $\pm 0.84$	69.28 $\pm 0.73$	68.71 $\pm 0.40$	56.01
TS	32.10 $\pm 1.18$	49.66 $\pm 0.64$	69.12 $\pm 0.48$	67.01 $\pm 0.39$	54.47
IR	32.67 $\pm 1.14$	51.26 $\pm 0.79$	68.83 $\pm 0.85$	67.58 $\pm 0.30$	55.09
HB	32.00 $\pm 1.01$	49.13 $\pm 0.57$	69.06 $\pm 0.82$	67.24 $\pm 1.13$	54.36
BayesianPRM	35.80 $\pm 0.56$	51.93 $\pm 0.77$	69.56 $\pm 0.33$	69.69 $\pm 0.90$	56.75

Appendix

Table 11: PRM-guided test-time scaling performance comparison with calibration sets. We compare Best-of-

N

(

N=16

) and Instance-Adaptive Scaling (IAS,

N_{\max}=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\uparrow

indicates that higher values are better.

\pm

denotes the standard deviation across three random training seeds.

	MathVision	MathVerse-VO	MathVista	MMStar	Average
Vanilla Ensemble PRM	34.88 $\pm 0.47$	50.21 $\pm 0.92$	68.26 $\pm 0.59$	68.03 $\pm 0.88$	55.35
BayesianPRM	35.86 $\pm 0.70$	51.40 $\pm 1.11$	69.40 $\pm 0.53$	69.33 $\pm 0.60$	56.50

Appendix

Table 12: Comparison with Vanilla Ensemble PRM . We compare Best-of-

N

(

N=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\pm

denotes the standard deviation across three random training seeds.

Figure 4: Reliability-aware belief distribution during training. We use

M=5

for visualization clarity.

	MathVision	MathVerse-VO	MathVista	MMStar	Average
$\beta_{1}=0.001$	34.28 $\pm 0.46$	50.30 $\pm 0.88$	68.33 $\pm 0.67$	68.56 $\pm 0.83$	55.37
$\beta_{1}=0.01$	35.67 $\pm 0.78$	51.86 $\pm 1.27$	69.32 $\pm 0.42$	69.07 $\pm 0.57$	56.48
$\beta_{1}=0.05$	35.86 $\pm 0.70$	51.40 $\pm 1.11$	69.40 $\pm 0.53$	69.33 $\pm 0.60$	56.50
$\beta_{1}=0.1$	35.21 $\pm 0.40$	50.35 $\pm 0.63$	68.82 $\pm 0.27$	68.19 $\pm 0.64$	55.64

Appendix

Table 13: Ablation Study on

\beta_{1}

. We compare Best-of-

N

(

N=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\pm

denotes the standard deviation across three random training seeds.

	MathVision	MathVerse-VO	MathVista	MMStar	Average
$\beta_{2}=0.001$	35.24 $\pm 0.62$	50.62 $\pm 0.94$	68.43 $\pm 0.46$	69.10 $\pm 0.82$	55.85
$\beta_{2}=0.01$	35.78 $\pm 0.76$	50.54 $\pm 0.85$	69.21 $\pm 0.50$	69.39 $\pm 0.74$	56.23
$\beta_{2}=0.05$	35.86 $\pm 0.70$	51.40 $\pm 1.11$	69.40 $\pm 0.53$	69.33 $\pm 0.60$	56.50
$\beta_{2}=0.1$	34.72 $\pm 0.49$	50.26 $\pm 1.26$	69.27 $\pm 0.61$	69.98 $\pm 0.85$	56.06

Appendix

Table 14: Ablation Study on

\beta_{2}

. We compare Best-of-

N

(

N=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\pm

denotes the standard deviation across three random training seeds.

	MathVision	MathVerse-VO	MathVista	MMStar	Average
$M=10$	35.30 $\pm 0.72$	51.42 $\pm 0.79$	68.97 $\pm 0.42$	68.49 $\pm 0.59$	56.05
$M=20$	35.86 $\pm 0.70$	51.40 $\pm 1.11$	69.40 $\pm 0.53$	69.33 $\pm 0.60$	56.50
$M=50$	35.89 $\pm 0.65$	51.35 $\pm 1.30$	69.52 $\pm 0.49$	68.99 $\pm 0.64$	56.44
$M=100$	35.87 $\pm 0.58$	51.46 $\pm 0.83$	69.34 $\pm 0.60$	69.62 $\pm 0.39$	56.57

Appendix

Table 15: Ablation Study on

M

. We compare Best-of-

N

(

N=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\pm

denotes the standard deviation across three random training seeds.

	MathVision	MathVerse-VO	MathVista	MMStar	Average
$\lambda_{\text{prior}}=0.1$	34.93 $\pm 0.49$	49.84 $\pm 0.53$	68.20 $\pm 0.73$	67.77 $\pm 0.43$	55.19
$\lambda_{\text{prior}}=1$	35.39 $\pm 0.79$	50.27 $\pm 0.88$	68.85 $\pm 0.56$	69.12 $\pm 0.93$	55.91
$\lambda_{\text{prior}}=10$	35.86 $\pm 0.70$	51.40 $\pm 1.11$	69.40 $\pm 0.53$	69.33 $\pm 0.60$	56.50
$\lambda_{\text{prior}}=50$	35.41 $\pm 1.09$	51.02 $\pm 0.69$	68.82 $\pm 0.64$	69.24 $\pm 0.48$	56.12

Appendix

Table 16: Ablation Study on

\lambda_{\text{prior}}

. We compare Best-of-

N

(

N=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\pm

denotes the standard deviation across three random training seeds.

MLLM Reasoning

See all 27 papers

May 11, 2026

Unsupervised Process Reward Models

Process Reward Models (PRMs) are a powerful mechanism for steering large language model reasoning by providing fine-grained, step-level supervision. However, this effectiveness comes at a significant cost: PRMs require expert annotations for every reasoning step, making them costly and difficult to scale. Here, we propose a method for training unsupervised PRMs (uPRM) that requires no human supervision, neither at the level of step-by-step annotations nor through ground-truth verification of final answers. The key idea behind our approach is to define a scoring function, derived from LLM next-token probabilities, that jointly assesses candidate positions of first erroneous steps across a batch of reasoning trajectories. We demonstrate the effectiveness of uPRM across diverse scenarios: (i) uPRM achieves up to 15% absolute accuracy improvements over the LLM-as-a-Judge in identifying first erroneous steps on the ProcessBench dataset; (ii) as a verifier for test-time scaling, uPRM performs comparably to supervised PRMs and outperforms the majority voting baseline by up to 6.9%, and (iii) when used as a reward signal in reinforcement learning, uPRM enables more robust policy optimization throughout training compared to a supervised PRM trained using ground-truth labels. Overall, our results open a path toward scalable reward modeling for complex reasoning tasks.

Artyom Gadetsky, Maxim Kodryan, Siba Smarak Panigrahi +2

Swiss Federal Institute of Technology (EPFL)

Reward Modeling Process Reward Models Reasoning Verification

Ask PDF

Dec 2, 2025

ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth

Training process reward models (PRMs) requires step-level correctness labels, obtained either through expensive human annotation or by relying on ground-truth answers, limiting the ability to scale process-level supervision. We propose ScalePRM, which scales verification compute as an alternative: given a problem and a candidate solution, we generate multiple independent verifications of each reasoning step and aggregate their judgments to produce synthetic step-level labels without ground truth. We explore two representative inference-time scaling strategies, parallel scaling through self-consistency and sequential scaling through meta-critique, and train generative PRMs on the resulting synthetic data. On ProcessBench, a benchmark for identifying erroneous steps in mathematical reasoning, PRMs trained on step-level self-consistency data achieve 67.5 F1, surpassing reference-guided training with ground-truth access (66.4 F1) and GPT-4o as a critic (61.9 F1). When deployed as reward signals in RL training with Qwen2.5-Math-7B, our best PRM achieves 47.4% average accuracy across six mathematical reasoning benchmarks, outperforming ground-truth-based RLVR (43.9%). We also identify and address reward exploitation patterns unique to generative PRM-based RL. Our results demonstrate that scaling verification compute is a viable alternative to ground-truth supervision for training process reward models.

Salman Rahman, Sruthi Gorantla, Arpit Gupta +3

2UCLA · Work done while as an intern at Amazon AGI. · 1Amazon AGI

Process Reward Models Reinforcement Learning with Verifiable Rewards LLM Mathematical Reasoning

Ask PDF

Jun 8, 2026

The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning

Process Reward Models (PRMs) improve credit assignment for reasoning by providing step-level feedback. However, we identify a hidden bias in PRMs caused by severe imbalance in step-level training data. Standard cross-entropy training amplifies this bias, causing PRMs to overcredit plausible but incorrect steps and produce high false-positive rates. We show that these false positives have an asymmetric downstream effect: false negatives mainly slow exploration, whereas false positives actively steer Best-of-N selection, guided decoding, and policy optimization toward flawed reasoning. This suggests that PRM training should shift from pointwise label fitting to reliable relative comparisons. To address this, we propose PRISM (Precision Ranking for Improved Step Modeling), a policy-aware PRM training framework that learns from contrastive step-level comparisons and hard negatives generated by a temporal lookahead strategy, requiring no new human labels. We further use a difficulty-aware curriculum to optimize the contrastive step margin. Across PRMBench and ProcessBench, PRISM substantially reduces false positives (22% on PRMBench) and improves macro F1 over strong discriminative PRMs. When applied to policy optimization and search tasks, including guided decoding and Best-of-N selection, it consistently improves accuracy (up to 22% for guided decoding and 33% for Best-of-N) and robustness. More broadly, trustworthy process supervision is not just about assigning high rewards, but about rewarding the right reasoning for the right reasons.

Aakriti Agrawal, Souradip Chakraborty, Armin Saghafian +6

University of Maryland, College Park · Capital One · University of Central Florida

Process Reward Models RL for Language Model Reasoning Step-Level Credit Assignment in RL Step-Level Credit Assignment

Ask PDF

Abstract

MLLM Reasoning

Explore similar work

Unsupervised Process Reward Models

ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth

The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning