Vision--language models (VLMs) are increasingly aligned via Group Relative Policy Optimization (GRPO)-style training. However, relying solely on terminal outcome rewards yields sparse credit assignment in multi-step reasoning, weakening the linkage between visual evidence and intermediate steps and often causing unstable optimization and visual hallucinations. We propose Differential Feedback, which automatically constructs token/step-level supervision masks by repairing erroneous reasoning trajectories, explicitly marking the key positions that require correction. Without costly large-scale step-by-step human annotations, our method enables process-level visual alignment and can be seamlessly integrated into existing GRPO-like frameworks. Experiments on multimodal reasoning benchmarks including MMMStar and MathVista show an average 3% improvement under matched compute budgets. Our approach offers an effective, low-cost solution for accurate vision--reasoning process alignment.
Figures & tables
Figure 1: Difference Feedback (DF) provides fine-grained process supervision for VLM alignment. When the policy produces an incorrect trajectory, a small-edit repair is generated; the difference between the two outputs yields a token-level mask that gates gradient updates.
Model
MathVista
MMStar
MMMU
MathVerse
MathVision
AI2D
InternVL3.5-8B-MPO [ 18 ]
75.9
–
71.2
54.8
52.6
–
InternVL3.5-8B-MPO (PRIME [ 7 ] )
78.5
–
73.6
61.9
56.9
–
InternVL3.5-8B-MPO (GRPO)
78.4
–
73.4
61.5
56.8
–
InternVL3.5-8B-MPO ( DF-GRPO )
79.9±1.1
–
74.8±0.6
63.5±0.9
58.2±1.0
–
Qwen2.5-VL-7B [ 2 ]
68.2
63.9
58.6
49.2
25.1
83.9
ThinkLite-7B [ 19 ]
74.3
63.7
53.1
52.2
29.9
83.0
Table 1: Main experimental results. For selected experiments, we report the mean over five random seeds together with the 95% bootstrap confidence interval (mean ± 95% CI). Improvements over the baseline methods are statistically significant under a paired bootstrap test ( p<0.01 ).
Method
MathVista
MMStar
MMMU
MathVerse
MathVision
AI2D
GSPO (w/o DF)
73.6
65.9
56.7
50.8
27.4
84.5
DF w/ LCS Align
75.9
67.5
57.1
52.3
29.7
84.6
DF w/ Soft Weight
75.5
67.6
57.2
53.5
30.3
85.4
DF w/ SFT-only Repair
73.3
65.1
56.5
50.5
26.2
84.3
DF w/o Min-Edit ( λ=0 )
75.4
67.0
56.9
52.1
28.1
84.8
DF w/o C1 (Image-Removal)
75.2
68.1
57.0
53.3
30.2
84.9
Table 2: Ablation study (7B). Impact of each component on multimodal reasoning and alignment performance. Accuracy (%), higher is better.
Vision-Language Models (VLMs) frequently suffer from visual perception errors and hallucinations that compromise answer accuracy in complex reasoning tasks. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising solution by optimizing policies using answer correctness signals. Despite their effectiveness, prevailing RLVR methods face two critical limitations. First, much of the sampling budget is wasted on trajectories doomed to fail due to early visual description errors. Second, sparse rewards cannot distinguish whether failures stem from visual perception or reasoning stages. We introduce MIRL, a decoupled framework that addresses both limitations by leveraging mutual information (MI) between generated descriptions and visual inputs as a cheap pre-screening signal. This enables intelligent budget allocation toward high-potential trajectories via forking, while decoupled training provides independent MI-based rewards for visual perception optimization, resolving reward blindness. Experiments on six vision-language reasoning benchmarks demonstrate that MIRL achieves 70.22% average accuracy and successfully surpasses the performance of sampling 16 complete trajectories using only 10 pre-samples with top-6 selection (25% fewer complete trajectories). Our code is available at: https://anonymous.4open.science/r/mirl-main/.
Yin Zhang, Jiaxuan Zhao, Zonghan Wu +5
School of Mathematics, Tianjin University, Tianjin, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Shanghai Advanced Institute of Finance (SAIFS), East China Normal University, Shanghai, China +4
Recent advancements in reinforcement learning with verifiable rewards (RLVR) have significantly improved the complex reasoning ability of vision-language models (VLMs). However, its outcome-level supervision is too coarse to diagnose and correct errors within the reasoning chain. To this end, we propose Perceval, a process reward model (PRM) that enables token-level error grounding, which can extract image-related claims from the response and compare them one by one with the visual evidence in the image, ultimately returning claims that contain perceptual errors. Perceval is trained with perception-intensive supervised training data. We then integrate Perceval into the RL training process to train the policy models. Specifically, compared to traditional GRPO, which applies sequence-level advantages, we apply token-level advantages by targeting penalties on hallucinated spans identified by Perceval, thus enabling fine-grained supervision signals. In addition to augmenting the training process, Perceval can also assist VLMs during the inference stage. Using Perceval, we can truncate the erroneous portions of the model's response, and then either have the model regenerate the response directly or induce the model to reflect on its previous output. This process can be repeated multiple times to achieve test-time scaling. Experiments show significant improvements on benchmarks from various domains across multiple reasoning VLMs trained with RL, highlighting the promise of perception-centric supervision as a general-purpose strategy. For test-time scaling, it also demonstrates consistent performance gains over other strategies, such as major voting. Our code and data will be publicly released at https://github.com/RUCAIBox/Perceval.
Yingqian Min, Kun Zhou, Yifan Li +6
Gaoling School of Artificial Intelligence, Renmin University of China. · Bytedance. · University of California, San Diego. +1
Visual question answering increasingly requires multi-step reasoning. Recent post-training with reinforcement learning under verifiable rewards (RLVR) and Group Relative Policy Optimization (GRPO) can improve multimodal reasoning, but most approaches rely on sparse outcome-only rewards. As a result, they struggle to tell whether an incorrect answer comes from a small mistake late in the reasoning or from an unhelpful trajectory from the start. A common solution is to train a process reward model (PRM) for step-level supervision, but this typically requires large-scale high-quality chain-of-thought annotations and additional training cost. We propose ProcessThinker, a practical post-training pipeline that provides step-level process rewards without training an explicit PRM. ProcessThinker first rewrites reasoning traces into a step-tagged format for cold-start supervised fine-tuning, then applies GRPO with a standard format reward and our rollout-based process reward. Concretely, for each intermediate step, we sample multiple continuations from that step and use the empirical success rate (final-answer verification) as the step reward. This gives dense credit assignment and encourages reasoning steps that more reliably support a correct conclusion, helping reduce inconsistent or self-contradictory progress across steps -- a key issue in logical reasoning. Across four challenging video benchmarks (Video-MMMU, MMVU, VideoMathQA, and LongVideoBench), ProcessThinker consistently improves over the baseline model Qwen3-VL-8B-Instruct
Jingpei Wu, Xiao Han, Weixiang Shen +3
LMU Munich · Harvard University · University of Cambridge +2