Process Reward Modeling with Reliability- and Conservatism-Aware Bayesian Belief

Zhongjian Qiao; Zhixiang Zhou; Jiafei Lyu; Peisong Wang; Qiaosheng Zhang; Siyang Gao; Shuang Qiu

cs.LGOct 16, 2026

Process Reward Modeling with Reliability- and Conservatism-Aware Bayesian Belief

Authors: Zhongjian Qiao, Zhixiang Zhou, Jiafei Lyu, Peisong Wang, Qiaosheng Zhang, Siyang Gao, Shuang Qiu

Organizations: CityUHK · Shanghai Innovation Institute · Tencent · Institute of Automation, Chinese Academy of Sciences

Abstract

Process Reward Models (PRMs) play a central role in test-time scaling for large language model (LLM) reasoning. However, PRMs may provide inaccurate process rewards for test-time scaling, which we identify as arising from two sources of uncertainty: (1) PRM uncertainty that arises from insufficient supervision data and imperfect PRM training; and (2) policy uncertainty that results from the mismatch between the policy to collect supervision data and the policy for test-time scaling, leading to systematic reward overestimation. Previous studies fail to jointly consider both uncertainties. To bridge this gap, we propose BayesianPRM, which treats the process reward as a random variable within a reward hypothesis space and maintains a Bayesian belief over it. Since exact Bayesian inference over the continuous hypothesis space is generally intractable, we discretize the reward hypothesis space with a reward ensemble and introduce two complementary belief updates, each targeting one source of uncertainty: (1) to address PRM uncertainty, we introduce reliability-aware belief updating that measures how well each ensemble member explains the supervision data and updates the belief with an amortized belief network optimized by the evidence lower bound (ELBO); and (2) to tackle policy uncertainty, we propose conservatism-aware belief calibration which further reweights the belief toward lower rewards to suppress overestimation. These two updates yield a reliability- and conservatism-aware Bayesian belief, whose expectation determines the process reward. Experiments on multimodal reasoning benchmarks indicate that BayesianPRM consistently yields more accurate process reward estimates while enhancing test-time scaling performance.

Figures & tables

	PRM Uncertainty	Policy Uncertainty	Technique
BetaPRM ( Li et al., 2026b )	✓	✗	Beta Distribution
QR-PRM ( Park et al., 2026 )	✗	✓	Quantile Regression
OT-PRM ( Ma et al., 2026 )	✗	✓	Optimal Transport
BayesianPRM (Ours)	✓	✓	Bayesian Learning

Table 1: Comparison between BayesianPRM and relevant recent works.

Panel of Figure 1 — Figure 1: Illustration of PRM uncertainty and policy uncertainty. (a)-(b) Prediction error distributions on MathV360K (ID) and Geometry3K (OOD), reflecting PRM uncertainty. (c)-(d) Prediction error distribution on UniGeo (ID) and MathVision (OOD), illustrating policy uncertainty. Prediction error is defined as the predicted reward minus the empirical success probability.

Brier Score $\downarrow$
	MathVision	MathVerse-VO	MathVista	MMStar	Average
Standard PRM	0.2025 $\pm 0.0194$	0.2274 $\pm 0.0143$	0.1704 $\pm 0.0082$	0.2357 $\pm 0.0236$	0.2090
BetaPRM	0.2377 $\pm 0.0221$	0.2489 $\pm 0.0163$	0.1712 $\pm 0.0148$	0.2427 $\pm 0.0121$	0.2251
DreamPRM	0.2461 $\pm 0.0292$	0.2498 $\pm 0.0180$	0.1904 $\pm 0.0064$	0.2342 $\pm 0.0208$	0.2301
BayesianPRM	0.1411 $\pm 0.0201$	0.1848 $\pm 0.0112$	0.1579 $\pm 0.0107$	0.2085 $\pm 0.0153$	0.1731
Positive Brier Score $\downarrow$

Table 2: PRM accuracy evaluation results. We evaluate Brier Score, Positive Brier Score, ECE, and AverageCE on four benchmarks: MathVision, the Vision Only subset of MathVerse (denoted as MathVerse-VO), MathVista, and MMStar.

\downarrow

indicates that lower values are better.

\pm

denotes the standard deviation across three random training seeds.

Best-of-N ( $N=16$ ) $\uparrow$
	MathVision	MathVerse-VO	MathVista	MMStar	Average
InternVL3-8B	29.93	45.05	65.90	66.27	51.79
Self-Consistency	33.22	47.46	68.00	68.60	54.32
Standard PRM	32.57 $\pm 0.78$	50.78 $\pm 1.17$	69.00 $\pm 1.10$	67.47 $\pm 0.61$	54.96
BetaPRM	34.31 $\pm 0.70$	51.85 $\pm 0.63$	68.66 $\pm 1.15$	67.77 $\pm 1.54$	55.65
DreamPRM	34.79 $\pm 1.01$	51.22 $\pm 0.64$	69.13 $\pm 0.65$	67.18 $\pm 1.12$	55.58

Table 3: PRM-guided test-time scaling performance comparison. We compare Best-of-

N

(

N=16

) and Instance-Adaptive Scaling (IAS,

N_{\max}=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\uparrow

indicates that higher values are better.

\pm

denotes the standard deviation across three random training seeds.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
Backbone	InternVL3-8B
Optimizer	AdamW ( Loshchilov and Hutter, 2017 )
Learning Rate	$1\times 10^{-5}$
Weight Decay	0.05
Learning Rate Scheduler	Cosine
Warmup Ratio	0.05

Appendix

Table 6: Hyperparameters for BayesianPRM.

	MathVision	MathVerse-VO	MathVista	MMStar	Average
Brier Score
Standard PRM	0.2540 $\pm 0.0128$	0.2695 $\pm 0.0210$	0.2223 $\pm 0.0143$	0.2506 $\pm 0.0088$	0.2491
BetaPRM	0.2854 $\pm 0.0259$	0.2901 $\pm 0.0220$	0.2267 $\pm 0.0166$	0.2540 $\pm 0.0192$	0.2641
DreamPRM	0.2869 $\pm 0.0234$	0.3221 $\pm 0.0202$	0.2363 $\pm 0.0051$	0.2588 $\pm 0.0120$	0.2760
BayesianPRM	0.2002 $\pm 0.0187$	0.2439 $\pm 0.0261$	0.2118 $\pm 0.0242$	0.2262 $\pm 0.0105$	0.2205
Positive Brier Score

Appendix

Table 7: PRM accuracy evaluation results for Qwen3VL-8B. We evaluate Brier Score, Positive Brier Score, ECE, and AverageCE on four benchmarks: MathVision, the Vision Only subset of MathVerse (denoted as MathVerse-VO), MathVista, and MMStar.

\downarrow

indicates that lower values are better.

\pm

denotes the standard deviation across three random training seeds.

Best-of- $N$ ( $N=16$ )
	MathVision	MathVerse-VO	MathVista	MMStar	Average
Qwen3VL-8B	20.07	50.51	70.30	56.67	49.39
Self-Consistency	22.04	54.19	73.20	59.33	52.19
Standard PRM	20.86 $\pm 0.36$	53.52 $\pm 0.64$	74.97 $\pm 1.04$	59.61 $\pm 0.27$	52.24
BetaPRM	21.14 $\pm 0.57$	53.39 $\pm 0.33$	74.79 $\pm 0.70$	60.07 $\pm 0.44$	52.35
DreamPRM	21.06 $\pm 0.62$	53.71 $\pm 0.20$	74.15 $\pm 1.22$	60.27 $\pm 0.65$	52.30

Appendix

Table 8: PRM-guided test-time scaling performance comparison. We compare Best-of-

N

(

N=16

) and Instance-Adaptive Scaling (IAS,

N_{\max}=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\uparrow

indicates that higher values are better.

\pm

denotes the standard deviation across three random training seeds.

Model	Overall	MathVision	MathVerse	MMMU	DynaMath	WeMath
Standard PRM	59.79 $\pm 0.41$	57.03 $\pm 0.16$	59.65 $\pm 0.26$	62.23 $\pm 0.42$	59.71 $\pm 0.24$	63.13 $\pm 0.60$
BetaPRM	60.83 $\pm 0.23$	57.83 $\pm 0.38$	60.40 $\pm 0.44$	60.59 $\pm 0.26$	61.33 $\pm 0.62$	63.47 $\pm 0.48$
DreamPRM	60.69 $\pm 0.32$	58.13 $\pm 0.47$	59.44 $\pm 0.29$	61.22 $\pm 0.54$	61.06 $\pm 0.15$	63.59 $\pm 0.27$
BayesianPRM	62.23 $\pm 0.29$	61.58 $\pm 0.54$	61.31 $\pm 0.31$	63.39 $\pm 0.36$	61.12 $\pm 0.47$	64.06 $\pm 0.22$

Appendix

Table 9: Evaluation results on VisualProcessBench. Each source column reports macro-F1 on the corresponding subset, while Overall denotes the micro-F1 computed over all non-neutral annotated steps.

\pm

denotes the standard deviation across three random training seeds.

Brier Score $\downarrow$
	MathVision	MathVerse-VO	MathVista	MMStar	Average
QR-PRM	0.1634 $\pm 0.0166$	0.1899 $\pm 0.0147$	0.1343 $\pm 0.0124$	0.1795 $\pm 0.0262$	0.1668
TS	0.2931 $\pm 0.0216$	0.2652 $\pm 0.0124$	0.2145 $\pm 0.0189$	0.2667 $\pm 0.0160$	0.2599
IR	0.2568 $\pm 0.0357$	0.2420 $\pm 0.0324$	0.2001 $\pm 0.0164$	0.2504 $\pm 0.0293$	0.2373
HB	0.3146 $\pm 0.0250$	0.2847 $\pm 0.0127$	0.2394 $\pm 0.0165$	0.2781 $\pm 0.0277$	0.2792
BayesianPRM	0.1035 $\pm 0.0142$	0.1422 $\pm 0.0169$	0.0946 $\pm 0.0125$	0.1521 $\pm 0.0178$	0.1231

Appendix

Table 10: PRM accuracy evaluation results with calibration sets. We evaluate Brier Score, Positive Brier Score, ECE, and AverageCE on four benchmarks: MathVision, the Vision Only subset of MathVerse (denoted as MathVerse-VO), MathVista, and MMStar. TS, IR, HB denote Temperature Scaling, Isotonic Regression, Histogram Binning, respectively.

\downarrow

indicates that lower values are better.

\pm

denotes the standard deviation across three random training seeds.

Best-of- $N$ ( $N=16$ ) $\uparrow$
	MathVision	MathVerse-VO	MathVista	MMStar	Average
QR-PRM	34.79 $\pm 0.67$	51.26 $\pm 0.84$	69.28 $\pm 0.73$	68.71 $\pm 0.40$	56.01
TS	32.10 $\pm 1.18$	49.66 $\pm 0.64$	69.12 $\pm 0.48$	67.01 $\pm 0.39$	54.47
IR	32.67 $\pm 1.14$	51.26 $\pm 0.79$	68.83 $\pm 0.85$	67.58 $\pm 0.30$	55.09
HB	32.00 $\pm 1.01$	49.13 $\pm 0.57$	69.06 $\pm 0.82$	67.24 $\pm 1.13$	54.36
BayesianPRM	35.80 $\pm 0.56$	51.93 $\pm 0.77$	69.56 $\pm 0.33$	69.69 $\pm 0.90$	56.75

Appendix

Table 11: PRM-guided test-time scaling performance comparison with calibration sets. We compare Best-of-

N

(

N=16

) and Instance-Adaptive Scaling (IAS,

N_{\max}=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\uparrow

indicates that higher values are better.

\pm

denotes the standard deviation across three random training seeds.

	MathVision	MathVerse-VO	MathVista	MMStar	Average
Vanilla Ensemble PRM	34.88 $\pm 0.47$	50.21 $\pm 0.92$	68.26 $\pm 0.59$	68.03 $\pm 0.88$	55.35
BayesianPRM	35.86 $\pm 0.70$	51.40 $\pm 1.11$	69.40 $\pm 0.53$	69.33 $\pm 0.60$	56.50

Appendix

Table 12: Comparison with Vanilla Ensemble PRM . We compare Best-of-

N

(

N=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\pm

denotes the standard deviation across three random training seeds.

Figure 4: Reliability-aware belief distribution during training. We use

M=5

for visualization clarity.

	MathVision	MathVerse-VO	MathVista	MMStar	Average
$\beta_{1}=0.001$	34.28 $\pm 0.46$	50.30 $\pm 0.88$	68.33 $\pm 0.67$	68.56 $\pm 0.83$	55.37
$\beta_{1}=0.01$	35.67 $\pm 0.78$	51.86 $\pm 1.27$	69.32 $\pm 0.42$	69.07 $\pm 0.57$	56.48
$\beta_{1}=0.05$	35.86 $\pm 0.70$	51.40 $\pm 1.11$	69.40 $\pm 0.53$	69.33 $\pm 0.60$	56.50
$\beta_{1}=0.1$	35.21 $\pm 0.40$	50.35 $\pm 0.63$	68.82 $\pm 0.27$	68.19 $\pm 0.64$	55.64

Appendix

Table 13: Ablation Study on

\beta_{1}

. We compare Best-of-

N

(

N=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\pm

denotes the standard deviation across three random training seeds.

	MathVision	MathVerse-VO	MathVista	MMStar	Average
$\beta_{2}=0.001$	35.24 $\pm 0.62$	50.62 $\pm 0.94$	68.43 $\pm 0.46$	69.10 $\pm 0.82$	55.85
$\beta_{2}=0.01$	35.78 $\pm 0.76$	50.54 $\pm 0.85$	69.21 $\pm 0.50$	69.39 $\pm 0.74$	56.23
$\beta_{2}=0.05$	35.86 $\pm 0.70$	51.40 $\pm 1.11$	69.40 $\pm 0.53$	69.33 $\pm 0.60$	56.50
$\beta_{2}=0.1$	34.72 $\pm 0.49$	50.26 $\pm 1.26$	69.27 $\pm 0.61$	69.98 $\pm 0.85$	56.06

Appendix

Table 14: Ablation Study on

\beta_{2}

. We compare Best-of-

N

(

N=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\pm

denotes the standard deviation across three random training seeds.

	MathVision	MathVerse-VO	MathVista	MMStar	Average
$M=10$	35.30 $\pm 0.72$	51.42 $\pm 0.79$	68.97 $\pm 0.42$	68.49 $\pm 0.59$	56.05
$M=20$	35.86 $\pm 0.70$	51.40 $\pm 1.11$	69.40 $\pm 0.53$	69.33 $\pm 0.60$	56.50
$M=50$	35.89 $\pm 0.65$	51.35 $\pm 1.30$	69.52 $\pm 0.49$	68.99 $\pm 0.64$	56.44
$M=100$	35.87 $\pm 0.58$	51.46 $\pm 0.83$	69.34 $\pm 0.60$	69.62 $\pm 0.39$	56.57

Appendix

Table 15: Ablation Study on

M

. We compare Best-of-

N

(

N=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\pm

denotes the standard deviation across three random training seeds.

	MathVision	MathVerse-VO	MathVista	MMStar	Average
$\lambda_{\text{prior}}=0.1$	34.93 $\pm 0.49$	49.84 $\pm 0.53$	68.20 $\pm 0.73$	67.77 $\pm 0.43$	55.19
$\lambda_{\text{prior}}=1$	35.39 $\pm 0.79$	50.27 $\pm 0.88$	68.85 $\pm 0.56$	69.12 $\pm 0.93$	55.91
$\lambda_{\text{prior}}=10$	35.86 $\pm 0.70$	51.40 $\pm 1.11$	69.40 $\pm 0.53$	69.33 $\pm 0.60$	56.50
$\lambda_{\text{prior}}=50$	35.41 $\pm 1.09$	51.02 $\pm 0.69$	68.82 $\pm 0.64$	69.24 $\pm 0.48$	56.12

Appendix

Table 16: Ablation Study on

\lambda_{\text{prior}}

. We compare Best-of-

N

(

N=16

) on MathVision, MathVerse-VO, MathVista and MMStar.

\pm

denotes the standard deviation across three random training seeds.

MLLM Reasoning

See all 27 papers

Unsupervised Process Reward Models
May 11, 2026Artyom Gadetsky, Maxim Kodryan, Siba Smarak Panigrahi +2Reward Modeling Process Reward Models
Swiss Federal Institute of Technology (EPFL)
ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth
Dec 2, 2025Salman Rahman, Sruthi Gorantla, Arpit Gupta +3Process Reward Models Reinforcement Learning with Verifiable Rewards
2UCLA · Work done while as an intern at Amazon AGI. · 1Amazon AGI
The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning
Jun 8, 2026Aakriti Agrawal, Souradip Chakraborty, Armin Saghafian +6Process Reward Models RL for Language Model Reasoning
University of Maryland, College Park · Capital One · University of Central Florida

May 11, 2026J/K move · Enter open · S save

Unsupervised Process Reward Models

Artyom Gadetsky, Maxim Kodryan, Siba Smarak Panigrahi, Hang Guo+1

Swiss Federal Institute of Technology (EPFL)

Reward Modeling Process Reward Models Reasoning Verification

Open paper AskPDF arXiv

Process Reward Models (PRMs) are a powerful mechanism for steering large language model reasoning by providing fine-grained, step-level supervision. However, this effectiveness comes at a significant cost: PRMs require expert annotations for every reasoning step, making them costly and difficult to scale. Here, we propose a method for training unsupervised PRMs (uPRM) that requires no human supervision, neither at the level of step-by-step annotations nor through ground-truth verification of final answers. The key idea behind our approach is to define a scoring function, derived from LLM next-token probabilities, that jointly assesses candidate positions of first erroneous steps across a batch of reasoning trajectories. We demonstrate the effectiveness of uPRM across diverse scenarios: (i) uPRM achieves up to 15% absolute accuracy improvements over the LLM-as-a-Judge in identifying first erroneous steps on the ProcessBench dataset; (ii) as a verifier for test-time scaling, uPRM performs comparably to supervised PRMs and outperforms the majority voting baseline by up to 6.9%, and (iii) when used as a reward signal in reinforcement learning, uPRM enables more robust policy optimization throughout training compared to a supervised PRM trained using ground-truth labels. Overall, our results open a path toward scalable reward modeling for complex reasoning tasks.

Abstract

MLLM Reasoning

Explore similar work

Unsupervised Process Reward Models

ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth

The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning