Efficient Process Reward Modeling via Contrastive Mutual Information

Nakyung Lee; Sangwoo Hong; Jungwoo Lee

cs.CLApr 12, 2026

Efficient Process Reward Modeling via Contrastive Mutual Information

Authors: Nakyung Lee, Sangwoo Hong, Jungwoo Lee

Organizations: Department of Electrical and Computer Engineering, Seoul National University · Department of Computer Science and Engineering, Konkuk University

Abstract

Recent research has devoted considerable effort to verifying the intermediate reasoning steps of chain-of-thought (CoT) trajectories using process reward models (PRMs) and other verifier models. However, training a PRM typically requires human annotators to assign reward scores to each reasoning step, which is both costly and time-consuming. Existing automated approaches, such as Monte Carlo (MC) estimation, also demand substantial computational resources due to repeated LLM rollouts. To overcome these limitations, we propose contrastive pointwise mutual information (CPMI), a novel automatic reward labeling method that leverages the model's internal probability to infer step-level supervision while significantly reducing the computational burden of annotating dataset. CPMI quantifies how much a reasoning step increases the mutual information between the step and the correct target answer relative to hard-negative alternatives. This contrastive signal serves as a proxy for the step's contribution to the final solution and yields a reliable reward. The experimental results show that CPMI-based labeling reduces dataset construction time by 84% and token generation by 98% compared to MC estimation, while achieving higher accuracy on process-level evaluations and mathematical reasoning benchmarks.

Figures & tables

Figure 1: Main framework of our reward modeling and PRM training. We sample both gold and hard-negative answer and compute the logit differences between the with-step and without-step through a simple forward pass for each answer. The CPMI reward for each step is defined as the difference between these logit gaps, quantifying how much the step shifts the model’s belief toward the gold answer while diverging from the wrong answer. After normalizing the CPMI-based rewards and using them as soft labels, we train the PRM with a binary cross-entropy objective. The trained PRM is subsequently used during inference to evaluate or rerank reasoning trajectories.

	Model Quality				Computation Cost (Ratio)		RelEff ( $\mathbf{\times}$ )
Reward Type	AUC	PB	PRMB	MATH	Time	Token	AUC	PB	PRMB	MATH
MC	0.759	27.7	38.8	45.4	1.00	1.00	1.00	1.00	1.00	1.00
PAV	0.757	36.6	49.6	47.2	1.17	2.38	0.85	1.12	1.09	0.89
CPMI	0.765	34.6	58.8	48.2	0.16 ( $\downarrow$ 84%)	0.02 ( $\downarrow$ 98%)	6.34	7.85	9.53	6.68
CPMI_Merge	0.766	36.8	60.7	49.4	0.30 ( $\downarrow$ 70%)	0.18 ( $\downarrow$ 82%)	3.38	4.45	5.24	3.64

Table 1: Efficiency comparison of CPMI-based reward methods with MC. Computation costs (Time, Generated Tokens) are accumulated over a subset of 10K samples. RelEff quantifies the quality–cost trade-off relative to the baseline, as illustrated in Eq. 9 . Best results are highlighted in bold , and second-best results are underlined .

Setting	AUC	PB	PRMB	Math	MMLU
Gold_only	0.419	15.53	41.85	41.6	71.52
Neg_only	0.725	24.65	57.42	40.8	68.92
Both	0.765	39.48	60.04	50.4	75.67

Table 2: Comparison of different reward signal configurations. Gold_only uses only the correct-answer term, Neg_only uses only the contrastive negative term, and Both combines both terms as in Eq. 3 . AUC measures the step-level discriminative ability in a threshold-free manner. PB denotes the average F1 score, PRMB denotes the overall PRMScore, and MATH refers to the MATH500 BoN@8 accuracy.

Reward Type	ROC-AUC ( $\uparrow$ )	Wasserstein ( $\uparrow$ )
MC	0.759	0.092
PAV	0.757	0.105
CPMI	0.765	0.135
CPMI_Merge	0.766	0.133
Rand_Merge	0.379	0.007

Table 3: Distribution-level analysis of PRM scores on the ProcessBench-Math split. Wasserstein reports the 1-Wasserstein distance between the two score distributions in probability space. Bold values indicate the best performance for each metric.

Figure 2: PRM probability distributions on ProcessBench Math split. The x-axis indicates the PRM probabilites. The y-axis shows the normalized density of step occurrences. Blue and orange histograms correspond to correct and incorrect reasoning steps, respectively.

Reward Type	GSM8K	Math	Olympiad	Omni	Average
MC	35.2	38.8	17.9	18.9	27.7
PAV	51.8	43.8	27.6	23.1	36.6
CPMI	49.4	39.6	24.5	24.8	34.6
CPMI_Merge	52.0	40.8	28.7	25.6	36.8
Rand_Merge	21.7	24.9	12.2	15.9	18.7
MS*	47.9	33.9	24.8	19.8	31.5

Table 4: Performance by reward type on ProcessBench (F1 only). The Average column indicates the mean F1 across the four datasets. Rows marked with * correspond to reported performance in Wang et al. (2024) . Bold values indicate the best F1 in each column.

Reward Type	Simplicity	Soundness	Sensitivity	Total
MC	40.88	36.92	35.06	38.80
PAV	47.16	49.56	47.20	49.64
CPMI	52.56	60.21	55.59	58.75
CPMI_Merge	54.65	62.30	56.50	60.67
Rand_Merge	27.65	21.64	18.65	23.20

Table 5: PRMBench results. Average scores over nine categories into three step-level dimensions (Simplicity, Soundness, Sensitivity) and the overall average for each reward method.

Figure 3: Best-of- $N$ accuracy on math benchmarks. The x-axis denotes the number of samples $N$ used for Best-of- $N$ selection, and the y-axis reports accuracy.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: The four prompts used for diversification.

Panel of Figure 5 — Figure 5: Prompt for generating hard-negative answers.

Statistic	Ratio (%) / Count
Total samples	80,000
Task Source (GSM8K : MATH)	$1:1$
Composition (mixed : pure)	$7:3$

Appendix

Table 6: Statistics of the CPMI-80k dataset. All values are reported as counts or ratios.

	GSM8K			Math			Olympiad			Omni			Average
Reward Type	Err	Corr	F1	Err	Corr	F1	Err	Corr	F1	Err	Corr	F1	F1
MC	$22.2$	$85.0$	$35.2$	$25.9$	$76.8$	$38.8$	$10.7$	$54.3$	$17.9$	$11.5$	$53.9$	$18.9$	27.7
PAV	$35.7$	$93.8$	$51.8$	$32.2$	$68.5$	43.8	$17.5$	$64.3$	$27.6$	$14.5$	$57.3$	$23.1$	36.6
CPMI	$35.7$	$79.8$	$49.4$	$29.0$	$62.8$	$39.6$	$15.9$	$53.7$	$24.5$	$17.5$	$42.3$	$24.8$	34.6
CPMI_Merge	$37.2$	$86.5$	52.0	$31.0$	$59.6$	$40.8$	$20.1$	$49.9$	28.7	$18.4$	$41.9$	25.6	36.8
Rand_Merge	$13.5$	$55.4$	$21.7$	$16.5$	$50.5$	$24.9$	$7.1$	$43.4$	$12.2$	$9.6$	$46.1$	$15.9$	18.7

Appendix

Table 7: Detailed Performance by reward type on ProcessBench. The last two rows marked with * correspond to the reported performance in Wang et al. (2024) . Bold values indicate the best F1 score in each column.

	Simplicity		Soundness				Sensitivity			Overall
Reward Type	NR	NCL	ES	SC	DC	CI	PS	DR	MS
MC	37.56	44.20	43.18	26.84	36.72	40.94	42.28	42.30	20.61	38.8
PAV	45.69	48.62	55.86	39.09	46.96	56.32	50.17	53.18	38.25	49.64
CPMI	50.22	54.89	63.76	56.99	56.58	63.52	57.70	62.02	47.06	58.75
CPMI_Merge	51.27	58.02	65.99	58.96	58.70	65.53	59.28	63.34	46.89	60.67
Rand_Merge	20.14	35.15	26.40	19.26	21.11	19.79	25.71	23.24	7.00	23.20

Appendix

Table 8: PRMBench results across reward designs. Bold values indicate the best PRMScore.

Labeler	Reward	ROC-AUC $\uparrow$	PB $\uparrow$	PRMB $\uparrow$	MATH500 $\uparrow$	$\Delta$ Time	$\Delta$ Token
Qwen2.5-1.5B	MC	0.72	28.4	16.2	41.6	0%	0%
Qwen2.5-1.5B	CPMI-based	0.75	36.1	51.8	45.8	-65%	-82%
Llama3.2-1B	MC	0.73	29.5	18.6	48.0	0%	0%
Llama3.2-1B	CPMI-based	0.76	37.0	59.1	47.6	-73%	-82%

Appendix

Table 9: Robustness of CPMI across smaller labeling models from different families.

Reward Type	FOLIO $\uparrow$	LogiQA 2.0 $\uparrow$	LogicNLI $\uparrow$
MC	54.19	33.39	25.4
CPMI	59.61	37.71	32.1
CPMI_Merge	58.62	35.66	30.8
Rand_Merge	42.86	24.54	24.5

Appendix

Table 10: Generalization to out-of-distribution logical reasoning benchmarks under zero-shot BoN@8.

Loss	Reward	ProcessBench	PRMBench	MATH500
BCE	MC	27.7	38.8	45.4
BCE	CPMI	34.6	58.8	48.2
PQM	MC	37.9	45.4	44.0
PQM	CPMI	35.6	56.7	46.2
MSE	MC	31.0	43.1	45.4
MSE	CPMI	35.5	59.9	47.2

Appendix

Table 11: Robustness of CPMI across different PRM training objectives.

Figure 6: Ablation results varying the number of hard-negative targets $M$ in the CPMI objective. We report both process-level metrics (ROC-AUC, PB, PRM) and downstream performance (Math, MMLU).

			Model Quality				Computation Cost (Ratio)			RelEff ( $\times$ )
Merge Index	MC(%)	CPMI(%)	ROC-AUC	PB	PRMB	MATH	Time	ratio	gen_tokens	AUC	PB	PRMB	MATH
CPMI	0	100	0.765	34.6	58.8	48.2	38,603	0.16	1,579,148	6.34	7.85	9.53	6.68
CPMI_Merge1	16.6	83.4	0.766	36.8	60.7	49.4	72,553	0.30	16,420,803	3.38	4.45	5.24	3.64
CPMI_Merge2	33.2	66.8	0.784	35.8	58.9	47.0	106,452	0.44	31,239,824	2.36	2.95	3.47	2.36
CPMI_Merge3	48.0	52.0	0.804	36.6	56.7	56.6	136,579	0.56	44,410,422	1.88	2.35	2.60	2.22
MC	100	0	0.759	27.7	38.8	45.4	242,930	1.00	90,902,162	1.00	1.00	1.00	1.00

Appendix

Table 12: Efficiency analysis of CPMI merging variants across quality, cost, and cost-effectiveness metrics.

	Model Quality				Computation Cost (Ratio)		RelEff ( $\mathbf{\times}$ )
Reward Type	AUC	PB	PRMB	MATH	Time	Token	AUC	PB	PRMB	MATH
PAV	0.757	36.6	49.6	47.2	1.00	1.00	1.00	1.00	1.00	1.00
CPMI	0.765	34.6	58.8	48.2	0.14 ( $\downarrow$ 86%)	0.01 ( $\downarrow$ 99%)	7.46	6.99	8.75	7.55
CPMI_Merge1	0.766	36.3	60.2	49.2	0.28 ( $\downarrow$ 72%)	0.17 ( $\downarrow$ 83%)	3.63	3.56	4.35	3.74
CPMI_Merge2	0.750	37.5	56.8	49.6	0.42 ( $\downarrow$ 58%)	0.34 ( $\downarrow$ 66%)	2.34	2.43	2.71	2.49
CPMI_Merge3	0.797	39.9	56.1	48.1	0.55 ( $\downarrow$ 45%)	0.48 ( $\downarrow$ 52%)	1.91	1.98	2.06	1.86

Appendix

Table 13: Efficiency comparison of CPMI and PAV. Computation costs (Time, Generated Tokens) are accumulated over subset of 10K samples.

Process Reward Models

Process reward models assess intermediate reasoning steps and actions to guide learning, search, and evaluation.

See all 140 papers

ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth
Dec 2, 2025Salman Rahman, Sruthi Gorantla, Arpit Gupta +3Process Reward Models Reinforcement Learning with Verifiable Rewards
2UCLA · Work done while as an intern at Amazon AGI. · 1Amazon AGI
Unsupervised Process Reward Models
May 11, 2026Artyom Gadetsky, Maxim Kodryan, Siba Smarak Panigrahi +2Reward Modeling Process Reward Models
Swiss Federal Institute of Technology (EPFL)
RAWR: Reward Assignment Without Rollouts in Verifiable Domains
Mar 18, 2026Corentin Royer, Anna Hedström, Debarun Bhattacharjya +3Process Reward Models Reinforcement Learning with Verifiable Rewards
International Business Machines · ETH AI Center · Lirio +1

Dec 2, 2025J/K move · Enter open · S save

ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth

Salman Rahman, Sruthi Gorantla, Arpit Gupta, Swastik Roy+2

2UCLA · Work done while as an intern at Amazon AGI. · 1Amazon AGI

Process Reward Models Reinforcement Learning with Verifiable Rewards LLM Mathematical Reasoning

Open paper AskPDF arXiv

Training process reward models (PRMs) requires step-level correctness labels, obtained either through expensive human annotation or by relying on ground-truth answers, limiting the ability to scale process-level supervision. We propose ScalePRM, which scales verification compute as an alternative: given a problem and a candidate solution, we generate multiple independent verifications of each reasoning step and aggregate their judgments to produce synthetic step-level labels without ground truth. We explore two representative inference-time scaling strategies, parallel scaling through self-consistency and sequential scaling through meta-critique, and train generative PRMs on the resulting synthetic data. On ProcessBench, a benchmark for identifying erroneous steps in mathematical reasoning, PRMs trained on step-level self-consistency data achieve 67.5 F1, surpassing reference-guided training with ground-truth access (66.4 F1) and GPT-4o as a critic (61.9 F1). When deployed as reward signals in RL training with Qwen2.5-Math-7B, our best PRM achieves 47.4% average accuracy across six mathematical reasoning benchmarks, outperforming ground-truth-based RLVR (43.9%). We also identify and address reward exploitation patterns unique to generative PRM-based RL. Our results demonstrate that scaling verification compute is a viable alternative to ground-truth supervision for training process reward models.

Abstract

Process Reward Models

Explore similar work

ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth

Unsupervised Process Reward Models

RAWR: Reward Assignment Without Rollouts in Verifiable Domains