Process Reward Models (PRMs) provide fine-grained supervision over intermediate reasoning steps, yet most existing models lack long-range reasoning and deep thinking capabilities. Although prior work has introduced Chain-of-Thought (CoT) to reward models, the high annotation cost of CoT-PRM still limits its scalability. To address these challenges, we propose the Visual Reasoning Process Reward Model (VRPRM) and design a label-efficient two-stage training strategy. Experimental results demonstrate that with only 5.5K CoT-PRM supervised step labels and 40K non-CoT PRM reinforcement learning training examples, VRPRM achieves a macro F1 score of 69.18 on VisualProcessBench, outperforming VisualPRM-8B, a non-thinking PRM trained on 400K examples, by 7.00 points. In Best-of-N evaluation, VRPRM-RL brings absolute Overall improvements of 21.77, 23.42, and 23.12 points on InternVL2.5-8B, 26B, and 38B policies, respectively, with a maximum relative improvement of 59.0%.
Figures & tables
Reward Model
PRM
MM
CoT
RL
RRM Guo et al. (2025)
✓
✓
RM-R1 Chen et al. (2025a)
✓
✓
Think-RM Hong et al. (2025)
✓
✓
R1-Reward Zhang et al. (2025a)
✓
✓
✓
UnifiedReward Wang et al. (2025c)
✓
✓
✓
Qwen-VL-PRM-7B Ong et al. (2025)
✓
✓
Table 1: Comparison of representative reward models. VRPRM combines process-level evaluation, multimodal inputs, explicit CoT reasoning, and RL training. PRM denotes process-level evaluation; MM , multimodality; CoT , explicit reasoning; and RL , reinforcement-learning-based training.
Figure 1: Why process-level rewards for RL. Top: a trajectory with sound intermediate steps but a wrong final answer. Left: outcome supervision reduces to one bit of ground truth, so the judge “knows it is wrong but not how,” and whole trajectories—including lucky guesses—receive undifferentiated rewards. Right: step-level ground truth supports graded rewards (0.4–1.0) that separate partial agreement from exact judgment and pinpoint the first error; in a binary outcome task, an ORM still rewards a random guess with probability 50%.
Figure 2: Overview of VRPRM. Left: dual-thresholded MCTS labels and two Kimi-K2.6 calls produce evaluation-aligned multi-turn CoT-PRM data; uncertain judgments remain as loss-masked history. Right: weighted SFT yields VRPRM-SFT, followed by group-based RL on 40K high-confidence non-CoT examples with structured step, format, and thinking-length rewards.
Model
# Samples
DynaMath
MMMU
MathVerse
MathVision
WeMath
Overall
Proprietary Models
GPT-4o-mini
–
56.57
54.08
52.53
51.42
56.74
53.57
GPT-4o
–
56.30
60.20
59.70
59.00
63.30
60.30
Doubao-Seed-1.6-Vision
–
66.19
59.47
63.12
61.07
62.74
62.77
Gemini-2.0-Flash
–
58.50
60.10
62.80
66.70
58.70
62.30
Gemini-2.5-Pro
–
68.47
63.34
68.26
65.15
69.48
67.13
Table 2: Macro F1 on VisualProcessBench. Qwen3-VL-8B-Thinking and the thinking variants share the stepwise single-token inference interface; “w/o Thinking” scores all steps in one pass with the same visual input; “+ Cont. SFT” continues supervised training from VRPRM-SFT on the same 40K pool. Bold and underlined values denote the best and second-best result in each column. The 5.5K count refers to trainable SFT step labels from 1,378 multi-turn conversations, and 40K denotes the second-stage non-CoT pool; “–” indicates not applicable or not reported.
Model
VPB F1
Time / Sample (s)
Output Tok. / Sample
Output Tok. / Step
Qwen3-VL-8B-Thinking
42.48
5.466
854.34
85.97
VRPRM-SFT
66.77
2.417
273.22
27.49
w/o Thinking
61.63
0.999
35.09
3.53
VRPRM-RL
69.18
2.115
242.06
24.36
w/o Thinking
65.47
0.551
35.00
3.52
VisualPRM-8B
62.18
0.405
9.94
1.00
Table 3: Macro F1 and computational overhead on VisualProcessBench. All models use the same timing and token-counting protocol. Output tokens include global thinking and step judgments. Bold marks the best value in each column; lower is better for latency and token counts.
Model
MathVista
MathVision
MathVerse-VO
WeMath
LogicVista
MMMU
Overall
Proprietary Models
GPT-4o
60.00
31.20
40.60
45.80
52.80
70.70
50.18
Gemini-2.0-Flash
70.40
43.60
47.80
47.40
52.30
69.90
55.23
Claude-3.5-Sonnet
65.30
35.60
46.30
44.00
60.40
66.40
53.00
Open-source Policies and PRM-Guided Selection
InternVL2.5-8B
58.70
18.95
26.65
30.00
35.12
52.00
36.90
Table 4: Best-of-8 results on six multimodal reasoning benchmarks. MathVerse uses the Vision-Only (VO) split, and Overall is the mean across all six benchmarks. Each Δ row reports the absolute change over the corresponding policy. Bold and underlined values denote the best and second-best result in each column.
Figure 3: Best-of- N results for InternVL2.5-8B on four benchmarks. Pass@K is the oracle upper bound; Major@K is majority voting.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Verbatim system prompts and user-input composition for the two Kimi-K2.6 teacher calls.
Figure 5: Evaluation-aligned multi-turn SFT format. Global thinking is generated once, and each subsequent assistant turn predicts exactly one step label.
Statistic
Count
Source-level conversations
1,378
train / validation conversations
1,364 / 14
Source annotation files / unique images
38 / 1,332
Negative / positive source samples
910 / 468
Confidence thresholds τ−/τ+
0.125 / 0.75
Global-thinking turns
1,378
Appendix
Table 5: Statistics of the final multi-turn SFT dataset. Class balance is computed over trainable step labels rather than source examples.
Figure 6: Statistics of the final multi-turn SFT dataset. (a) Composition by source polarity, step supervision status, and trainable label class. (b) Distributions of all and trainable steps per conversation. (c) Character-length distribution of the global-thinking targets.
RL dataset statistic
Value
Training / validation examples
40,000 / 2,000
Negative / positive source examples
22,658 / 17,342
Negative / positive training steps
52,774 / 112,227
Total training step labels
165,001
Negative-step ratio
31.98%
Steps per example
1–24
Appendix
Table 6: Statistics of the confidence-filtered and balanced RL data. Source polarity and step-label polarity are reported separately.
Figure 7: RL user prompt, system instruction, and constrained rollout structure. The output contains exactly one global-thinking block and one answer block with indexed binary step labels.
Figure 8: Two-stage VRPRM training dynamics. (a–b) SFT loss and token accuracy. (c–f) Primary rewards, class-wise F1, auxiliary rewards, and response behavior over the complete 600-step RL run. Faint traces are raw batch values; bold traces are causal seven-step moving averages for SFT and causal 21-step moving averages for RL. Circles denote SFT validation, and dashed diamond curves denote RL validation every 50 steps.
Figure 9: Visual-logic input used in the VRPRM scoring example.
Candidate 1 (Correct)
Candidate 2 (Incorrect)
1. Policy Generated Step List
Step 1: Infer the separating rule from the example flowers in Sets A and B. Step 2: Each Set A flower has four shaded petals, whereas each Set B flower has five. Step 3: The query flower has five shaded petals. Step 4: It therefore follows the rule of Set B. Step 5: Answer: B
Step 1: Infer the separating rule from the example flowers in Sets A and B. Step 2: Each Set A flower has four shaded petals, whereas each Set B flower has five. Step 3: The query flower has four shaded petals. Step 4: It therefore follows the rule of Set A. Step 5: Answer: A
2. Global Thinking and Binary Step Judgments
<think > The separating rule is the number of shaded petals: Set A has four and Set B has five. The query has five, so the candidate’s visual count and conclusion are supported. </think > Judgments: Step 1 1 ; Step 2 1 ; Step 3 1 ; Step 4 1 ; Step 5 1 .
<think > The candidate derives the set rule correctly but miscounts the query in Step 3: it has five shaded petals, not four. Steps 4 and 5 depend on this visual error and are therefore invalid. </think > Judgments: Step 1 1 ; Step 2 1 ; Step 3 0 ; Step 4 0 ; Step 5 0 .
Table 7: VRPRM scoring on a visual-logic task. All steps of the correct candidate receive score 1.0. For the incorrect candidate, the score drops to near zero at Step 3, where the shaded petals are first miscounted, and remains low for the dependent conclusions.
Visual question answering increasingly requires multi-step reasoning. Recent post-training with reinforcement learning under verifiable rewards (RLVR) and Group Relative Policy Optimization (GRPO) can improve multimodal reasoning, but most approaches rely on sparse outcome-only rewards. As a result, they struggle to tell whether an incorrect answer comes from a small mistake late in the reasoning or from an unhelpful trajectory from the start. A common solution is to train a process reward model (PRM) for step-level supervision, but this typically requires large-scale high-quality chain-of-thought annotations and additional training cost. We propose ProcessThinker, a practical post-training pipeline that provides step-level process rewards without training an explicit PRM. ProcessThinker first rewrites reasoning traces into a step-tagged format for cold-start supervised fine-tuning, then applies GRPO with a standard format reward and our rollout-based process reward. Concretely, for each intermediate step, we sample multiple continuations from that step and use the empirical success rate (final-answer verification) as the step reward. This gives dense credit assignment and encourages reasoning steps that more reliably support a correct conclusion, helping reduce inconsistent or self-contradictory progress across steps -- a key issue in logical reasoning. Across four challenging video benchmarks (Video-MMMU, MMVU, VideoMathQA, and LongVideoBench), ProcessThinker consistently improves over the baseline model Qwen3-VL-8B-Instruct
Jingpei Wu, Xiao Han, Weixiang Shen +3
LMU Munich · Harvard University · University of Cambridge +2
Process Reward Models (PRMs) are a powerful mechanism for steering large language model reasoning by providing fine-grained, step-level supervision. However, this effectiveness comes at a significant cost: PRMs require expert annotations for every reasoning step, making them costly and difficult to scale. Here, we propose a method for training unsupervised PRMs (uPRM) that requires no human supervision, neither at the level of step-by-step annotations nor through ground-truth verification of final answers. The key idea behind our approach is to define a scoring function, derived from LLM next-token probabilities, that jointly assesses candidate positions of first erroneous steps across a batch of reasoning trajectories. We demonstrate the effectiveness of uPRM across diverse scenarios: (i) uPRM achieves up to 15% absolute accuracy improvements over the LLM-as-a-Judge in identifying first erroneous steps on the ProcessBench dataset; (ii) as a verifier for test-time scaling, uPRM performs comparably to supervised PRMs and outperforms the majority voting baseline by up to 6.9%, and (iii) when used as a reward signal in reinforcement learning, uPRM enables more robust policy optimization throughout training compared to a supervised PRM trained using ground-truth labels. Overall, our results open a path toward scalable reward modeling for complex reasoning tasks.
Training process reward models (PRMs) requires step-level correctness labels, obtained either through expensive human annotation or by relying on ground-truth answers, limiting the ability to scale process-level supervision. We propose ScalePRM, which scales verification compute as an alternative: given a problem and a candidate solution, we generate multiple independent verifications of each reasoning step and aggregate their judgments to produce synthetic step-level labels without ground truth. We explore two representative inference-time scaling strategies, parallel scaling through self-consistency and sequential scaling through meta-critique, and train generative PRMs on the resulting synthetic data. On ProcessBench, a benchmark for identifying erroneous steps in mathematical reasoning, PRMs trained on step-level self-consistency data achieve 67.5 F1, surpassing reference-guided training with ground-truth access (66.4 F1) and GPT-4o as a critic (61.9 F1). When deployed as reward signals in RL training with Qwen2.5-Math-7B, our best PRM achieves 47.4% average accuracy across six mathematical reasoning benchmarks, outperforming ground-truth-based RLVR (43.9%). We also identify and address reward exploitation patterns unique to generative PRM-based RL. Our results demonstrate that scaling verification compute is a viable alternative to ground-truth supervision for training process reward models.
Salman Rahman, Sruthi Gorantla, Arpit Gupta +3
2UCLA · Work done while as an intern at Amazon AGI. · 1Amazon AGI