Difference Feedback: Generating Multimodal Process-Level Supervision for VLM Reinforcement Learning

Feiding; Yongkang Zhang; Yuhao Liao; Zijian Zeng; Chunzheng Zhu; Yaozong Zheng; Yafei Liu; Yeling Peng

cs.CVMar 29, 2026

Difference Feedback: Generating Multimodal Process-Level Supervision for VLM Reinforcement Learning

Authors: Feiding, Yongkang Zhang, Yuhao Liao, Zijian Zeng, Chunzheng Zhu, Yaozong Zheng, Yafei Liu, Yeling Peng,

Organizations: Shenzhen International Graduate School, Tsinghua University, China

Abstract

Vision--language models (VLMs) are increasingly aligned via Group Relative Policy Optimization (GRPO)-style training. However, relying solely on terminal outcome rewards yields sparse credit assignment in multi-step reasoning, weakening the linkage between visual evidence and intermediate steps and often causing unstable optimization and visual hallucinations. We propose Differential Feedback, which automatically constructs token/step-level supervision masks by repairing erroneous reasoning trajectories, explicitly marking the key positions that require correction. Without costly large-scale step-by-step human annotations, our method enables process-level visual alignment and can be seamlessly integrated into existing GRPO-like frameworks. Experiments on multimodal reasoning benchmarks including MMMStar and MathVista show an average 3% improvement under matched compute budgets. Our approach offers an effective, low-cost solution for accurate vision--reasoning process alignment.

Figures & tables

Figure 1: Difference Feedback (DF) provides fine-grained process supervision for VLM alignment. When the policy produces an incorrect trajectory, a small-edit repair is generated; the difference between the two outputs yields a token-level mask that gates gradient updates.

Model	MathVista	MMStar	MMMU	MathVerse	MathVision	AI2D
InternVL3.5-8B-MPO [ 18 ]	75.9	–	71.2	54.8	52.6	–
InternVL3.5-8B-MPO (PRIME [ 7 ] )	78.5	–	73.6	61.9	56.9	–
InternVL3.5-8B-MPO (GRPO)	78.4	–	73.4	61.5	56.8	–
InternVL3.5-8B-MPO ( DF-GRPO )	$79.9_{\pm 1.1}$	–	$74.8_{\pm 0.6}$	$63.5_{\pm 0.9}$	$58.2_{\pm 1.0}$	–
Qwen2.5-VL-7B [ 2 ]	68.2	63.9	58.6	49.2	25.1	83.9
ThinkLite-7B [ 19 ]	74.3	63.7	53.1	52.2	29.9	83.0

Table 1: Main experimental results. For selected experiments, we report the mean over five random seeds together with the 95% bootstrap confidence interval (mean

\pm

95% CI). Improvements over the baseline methods are statistically significant under a paired bootstrap test (

p<0.01

Method	MathVista	MMStar	MMMU	MathVerse	MathVision	AI2D
GSPO (w/o DF)	73.6	65.9	56.7	50.8	27.4	84.5
DF w/ LCS Align	75.9	67.5	57.1	52.3	29.7	84.6
DF w/ Soft Weight	75.5	67.6	57.2	53.5	30.3	85.4
DF w/ SFT-only Repair	73.3	65.1	56.5	50.5	26.2	84.3
DF w/o Min-Edit ( $\lambda=0$ )	75.4	67.0	56.9	52.1	28.1	84.8
DF w/o C1 (Image-Removal)	75.2	68.1	57.0	53.3	30.2	84.9

Table 2: Ablation study (7B). Impact of each component on multimodal reasoning and alignment performance. Accuracy (%), higher is better.

Table 4

RL for VLMs

Reinforcement learning for vision-language models aligns multimodal reasoning, tool use, and action with task-specific feedback.

See all 446 papers

MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Models
May 2, 2026Yin Zhang, Jiaxuan Zhao, Zonghan Wu +5VLM Reasoning Vision-Language Models
School of Mathematics, Tianjin University, Tianjin, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Shanghai Advanced Institute of Finance (SAIFS), East China Normal University, Shanghai, China +4
Improving Vision-language Models with Perception-centric Process Reward Models
Apr 27, 2026Yingqian Min, Kun Zhou, Yifan Li +6Process Reward Models Object Hallucination in VLMs
Gaoling School of Artificial Intelligence, Renmin University of China. · Bytedance. · University of California, San Diego. +1
ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward
Apr 23, 2026Jingpei Wu, Xiao Han, Weixiang Shen +3Process Reward Models Multimodal QA
LMU Munich · Harvard University · University of Cambridge +2

May 2, 2026J/K move · Enter open · S save

MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Models

Yin Zhang, Jiaxuan Zhao, Zonghan Wu, Zengxiang Li+4

School of Mathematics, Tianjin University, Tianjin, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Shanghai Advanced Institute of Finance (SAIFS), East China Normal University, Shanghai, China+4

VLM Reasoning Vision-Language Models Reinforcement Learning RL for VLMs Reinforcement Learning with Verifiable Rewards Visually Grounded Reasoning+1

Open paper AskPDF arXiv

Vision-Language Models (VLMs) frequently suffer from visual perception errors and hallucinations that compromise answer accuracy in complex reasoning tasks. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising solution by optimizing policies using answer correctness signals. Despite their effectiveness, prevailing RLVR methods face two critical limitations. First, much of the sampling budget is wasted on trajectories doomed to fail due to early visual description errors. Second, sparse rewards cannot distinguish whether failures stem from visual perception or reasoning stages. We introduce MIRL, a decoupled framework that addresses both limitations by leveraging mutual information (MI) between generated descriptions and visual inputs as a cheap pre-screening signal. This enables intelligent budget allocation toward high-potential trajectories via forking, while decoupled training provides independent MI-based rewards for visual perception optimization, resolving reward blindness. Experiments on six vision-language reasoning benchmarks demonstrate that MIRL achieves 70.22% average accuracy and successfully surpasses the performance of sampling 16 complete trajectories using only 10 pre-samples with top-6 selection (25% fewer complete trajectories). Our code is available at: https://anonymous.4open.science/r/mirl-main/.

Abstract

RL for VLMs

Explore similar work

MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Models

Improving Vision-language Models with Perception-centric Process Reward Models

ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward