VRPRM: Process Reward Modeling via Visual Reasoning

Xinquan Chen; Chongying Yue; Bangwei Liu; Xuhong Wang; Yingchun Wang; Chaochao Lu

cs.LGAug 5, 2025

VRPRM: Process Reward Modeling via Visual Reasoning

Authors: Xinquan Chen, Chongying Yue, Bangwei Liu, Xuhong Wang, Yingchun Wang, Chaochao Lu

Organizations: Shanghai AI Laboratory · Zhejiang University · East China Normal University

Abstract

Process Reward Models (PRMs) provide fine-grained supervision over intermediate reasoning steps, yet most existing models lack long-range reasoning and deep thinking capabilities. Although prior work has introduced Chain-of-Thought (CoT) to reward models, the high annotation cost of CoT-PRM still limits its scalability. To address these challenges, we propose the Visual Reasoning Process Reward Model (VRPRM) and design a label-efficient two-stage training strategy. Experimental results demonstrate that with only 5.5K CoT-PRM supervised step labels and 40K non-CoT PRM reinforcement learning training examples, VRPRM achieves a macro F1 score of 69.18 on VisualProcessBench, outperforming VisualPRM-8B, a non-thinking PRM trained on 400K examples, by 7.00 points. In Best-of-N evaluation, VRPRM-RL brings absolute Overall improvements of 21.77, 23.42, and 23.12 points on InternVL2.5-8B, 26B, and 38B policies, respectively, with a maximum relative improvement of 59.0%.

Figures & tables

Reward Model	PRM	MM	CoT	RL
RRM Guo et al. (2025)			✓	✓
RM-R1 Chen et al. (2025a)			✓	✓
Think-RM Hong et al. (2025)			✓	✓
R1-Reward Zhang et al. (2025a)		✓	✓	✓
UnifiedReward Wang et al. (2025c)		✓	✓	✓
Qwen-VL-PRM-7B Ong et al. (2025)	✓	✓

Table 1: Comparison of representative reward models. VRPRM combines process-level evaluation, multimodal inputs, explicit CoT reasoning, and RL training. PRM denotes process-level evaluation; MM , multimodality; CoT , explicit reasoning; and RL , reinforcement-learning-based training.

Figure 1: Why process-level rewards for RL. Top: a trajectory with sound intermediate steps but a wrong final answer. Left: outcome supervision reduces to one bit of ground truth, so the judge “knows it is wrong but not how,” and whole trajectories—including lucky guesses—receive undifferentiated rewards. Right: step-level ground truth supports graded rewards (0.4–1.0) that separate partial agreement from exact judgment and pinpoint the first error; in a binary outcome task, an ORM still rewards a random guess with probability 50%.

Figure 2: Overview of VRPRM. Left: dual-thresholded MCTS labels and two Kimi-K2.6 calls produce evaluation-aligned multi-turn CoT-PRM data; uncertain judgments remain as loss-masked history. Right: weighted SFT yields VRPRM-SFT, followed by group-based RL on 40K high-confidence non-CoT examples with structured step, format, and thinking-length rewards.

Proprietary Models
Model	# Samples	DynaMath	MMMU	MathVerse	MathVision	WeMath	Overall
GPT-4o-mini	–	56.57	54.08	52.53	51.42	56.74	53.57
GPT-4o	–	56.30	60.20	59.70	59.00	63.30	60.30
Doubao-Seed-1.6-Vision	–	66.19	59.47	63.12	61.07	62.74	62.77
Gemini-2.0-Flash	–	58.50	60.10	62.80	66.70	58.70	62.30
Gemini-2.5-Pro	–	68.47	63.34	68.26	65.15	69.48	67.13

Table 2: Macro F1 on VisualProcessBench. Qwen3-VL-8B-Thinking and the thinking variants share the stepwise single-token inference interface; “w/o Thinking” scores all steps in one pass with the same visual input; “+ Cont. SFT” continues supervised training from VRPRM-SFT on the same 40K pool. Bold and underlined values denote the best and second-best result in each column. The 5.5K count refers to trainable SFT step labels from 1,378 multi-turn conversations, and 40K denotes the second-stage non-CoT pool; “–” indicates not applicable or not reported.

Model	VPB F1	Time / Sample (s)	Output Tok. / Sample	Output Tok. / Step
Qwen3-VL-8B-Thinking	42.48	5.466	854.34	85.97
VRPRM-SFT	66.77	2.417	273.22	27.49
w/o Thinking	61.63	0.999	35.09	3.53
VRPRM-RL	69.18	2.115	242.06	24.36
w/o Thinking	65.47	0.551	35.00	3.52
VisualPRM-8B	62.18	0.405	9.94	1.00

Table 3: Macro F1 and computational overhead on VisualProcessBench. All models use the same timing and token-counting protocol. Output tokens include global thinking and step judgments. Bold marks the best value in each column; lower is better for latency and token counts.

Proprietary Models
Model	MathVista	MathVision	MathVerse-VO	WeMath	LogicVista	MMMU	Overall
GPT-4o	60.00	31.20	40.60	45.80	52.80	70.70	50.18
Gemini-2.0-Flash	70.40	43.60	47.80	47.40	52.30	69.90	55.23
Claude-3.5-Sonnet	65.30	35.60	46.30	44.00	60.40	66.40	53.00
Open-source Policies and PRM-Guided Selection
InternVL2.5-8B	58.70	18.95	26.65	30.00	35.12	52.00	36.90

Table 4: Best-of-8 results on six multimodal reasoning benchmarks. MathVerse uses the Vision-Only (VO) split, and Overall is the mean across all six benchmarks. Each

\Delta

row reports the absolute change over the corresponding policy. Bold and underlined values denote the best and second-best result in each column.

Figure 3: Best-of-

N

results for InternVL2.5-8B on four benchmarks. Pass@K is the oracle upper bound; Major@K is majority voting.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Verbatim system prompts and user-input composition for the two Kimi-K2.6 teacher calls.

Panel of Figure 5 — Figure 5: Evaluation-aligned multi-turn SFT format. Global thinking is generated once, and each subsequent assistant turn predicts exactly one step label.

Statistic	Count
Source-level conversations	1,378
train / validation conversations	1,364 / 14
Source annotation files / unique images	38 / 1,332
Negative / positive source samples	910 / 468
Confidence thresholds $\tau_{-}/\tau_{+}$	0.125 / 0.75
Global-thinking turns	1,378

Appendix

Table 5: Statistics of the final multi-turn SFT dataset. Class balance is computed over trainable step labels rather than source examples.

Figure 6: Statistics of the final multi-turn SFT dataset. (a) Composition by source polarity, step supervision status, and trainable label class. (b) Distributions of all and trainable steps per conversation. (c) Character-length distribution of the global-thinking targets.

RL dataset statistic	Value
Training / validation examples	40,000 / 2,000
Negative / positive source examples	22,658 / 17,342
Negative / positive training steps	52,774 / 112,227
Total training step labels	165,001
Negative-step ratio	31.98%
Steps per example	1–24

Appendix

Table 6: Statistics of the confidence-filtered and balanced RL data. Source polarity and step-label polarity are reported separately.

Panel of Figure 7 — Figure 7: RL user prompt, system instruction, and constrained rollout structure. The output contains exactly one global-thinking block and one answer block with indexed binary step labels.

Figure 8: Two-stage VRPRM training dynamics. (a–b) SFT loss and token accuracy. (c–f) Primary rewards, class-wise F1, auxiliary rewards, and response behavior over the complete 600-step RL run. Faint traces are raw batch values; bold traces are causal seven-step moving averages for SFT and causal 21-step moving averages for RL. Circles denote SFT validation, and dashed diamond curves denote RL validation every 50 steps.

Figure 9: Visual-logic input used in the VRPRM scoring example.

Candidate 1 (Correct)	Candidate 2 (Incorrect)
1. Policy Generated Step List
Step 1: Infer the separating rule from the example flowers in Sets A and B. Step 2: Each Set A flower has four shaded petals, whereas each Set B flower has five. Step 3: The query flower has five shaded petals. Step 4: It therefore follows the rule of Set B. Step 5: Answer: $\boxed{B}$	Step 1: Infer the separating rule from the example flowers in Sets A and B. Step 2: Each Set A flower has four shaded petals, whereas each Set B flower has five. Step 3: The query flower has four shaded petals. Step 4: It therefore follows the rule of Set A. Step 5: Answer: $\boxed{A}$
2. Global Thinking and Binary Step Judgments
<think > The separating rule is the number of shaded petals: Set A has four and Set B has five. The query has five, so the candidate’s visual count and conclusion are supported. </think > Judgments: Step 1 $\boxed{1}$ ; Step 2 $\boxed{1}$ ; Step 3 $\boxed{1}$ ; Step 4 $\boxed{1}$ ; Step 5 $\boxed{1}$ .	<think > The candidate derives the set rule correctly but miscounts the query in Step 3: it has five shaded petals, not four. Steps 4 and 5 depend on this visual error and are therefore invalid. </think > Judgments: Step 1 $\boxed{1}$ ; Step 2 $\boxed{1}$ ; Step 3 $\boxed{0}$ ; Step 4 $\boxed{0}$ ; Step 5 $\boxed{0}$ .
3. VRPRM Step Rewards ( $P_{\theta}(1\mid\text{context})$ )
Step 1: 1.0 Step 2: 1.0 Step 3: 1.0 Step 4: 1.0 Step 5: 1.0 Avg: 1.0 (Preferred)	Step 1: 1.0 Step 2: 1.0 Step 3: $\mathbf{5.22{\times}10^{-5}}$ $\downarrow$ First error Step 4: $1.81{\times}10^{-7}$ Step 5: $3.29{\times}10^{-6}$ Avg: 0.40 (Not Preferred)

Appendix

Table 7: VRPRM scoring on a visual-logic task. All steps of the correct candidate receive score 1.0. For the incorrect candidate, the score drops to near zero at Step 3, where the shaded petals are first miscounted, and remains low for the dependent conclusions.

Process Reward Models

Process reward models assess intermediate reasoning steps and actions to guide learning, search, and evaluation.

See all 140 papers

ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward
Apr 23, 2026Jingpei Wu, Xiao Han, Weixiang Shen +3Multimodal Understanding Multimodal QA
LMU Munich · Harvard University · University of Cambridge +2
Unsupervised Process Reward Models
May 11, 2026Artyom Gadetsky, Maxim Kodryan, Siba Smarak Panigrahi +2Reward Modeling Process Reward Models
Swiss Federal Institute of Technology (EPFL)
ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth
Dec 2, 2025Salman Rahman, Sruthi Gorantla, Arpit Gupta +3Process Reward Models Reinforcement Learning with Verifiable Rewards
2UCLA · Work done while as an intern at Amazon AGI. · 1Amazon AGI

Apr 23, 2026J/K move · Enter open · S save

ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward

Jingpei Wu, Xiao Han, Weixiang Shen, Boer Zhang+2

LMU Munich · Harvard University · University of Cambridge+2

Multimodal Understanding Multimodal QA Video Reasoning Multimodal CoT Reasoning Multimodal Foundation Models RL for VLMs+3

Open paper AskPDF arXiv

Visual question answering increasingly requires multi-step reasoning. Recent post-training with reinforcement learning under verifiable rewards (RLVR) and Group Relative Policy Optimization (GRPO) can improve multimodal reasoning, but most approaches rely on sparse outcome-only rewards. As a result, they struggle to tell whether an incorrect answer comes from a small mistake late in the reasoning or from an unhelpful trajectory from the start. A common solution is to train a process reward model (PRM) for step-level supervision, but this typically requires large-scale high-quality chain-of-thought annotations and additional training cost. We propose ProcessThinker, a practical post-training pipeline that provides step-level process rewards without training an explicit PRM. ProcessThinker first rewrites reasoning traces into a step-tagged format for cold-start supervised fine-tuning, then applies GRPO with a standard format reward and our rollout-based process reward. Concretely, for each intermediate step, we sample multiple continuations from that step and use the empirical success rate (final-answer verification) as the step reward. This gives dense credit assignment and encourages reasoning steps that more reliably support a correct conclusion, helping reduce inconsistent or self-contradictory progress across steps -- a key issue in logical reasoning. Across four challenging video benchmarks (Video-MMMU, MMVU, VideoMathQA, and LongVideoBench), ProcessThinker consistently improves over the baseline model Qwen3-VL-8B-Instruct

Abstract

Process Reward Models

Explore similar work

ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward

Unsupervised Process Reward Models

ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth