Reinforcement Learning with Visually Attributable On-Policy Self-Distillation

Shiao Xie; Siyu Chen; Jianwei Lv; Xiandong Li; Yujin Wang

cs.CVOct 19, 2026

Reinforcement Learning with Visually Attributable On-Policy Self-Distillation

Authors: Shiao Xie, Siyu Chen, Jianwei Lv, Xiandong Li, Yujin Wang

Organizations: Baidu Inc., China

Abstract

Multimodal reinforcement learning with verifiable rewards (RLVR) provides reliable task-level supervision, yet offers limited guidance for localizing errors in visual reasoning. On-policy self-distillation (OPSD) complements this supervision with dense token-level corrections from a privileged teacher equipped with an evidence-centered visual view. However, the resulting corrections may conflate visual evidence with non-visual shifts induced by linguistic priors, response style, or the privileged view itself. We formalize this phenomenon as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence. Directly distilling contaminated corrections may therefore reinforce erroneous token-level updates. To address this issue, we propose reinforcement learning with visually attributable on-policy self-distillation, an evidence-consistent paradigm that augments task-level RLVR supervision with privileged token-level corrections grounded in visual evidence. We instantiate this paradigm with Visual-Attribution Gating (VAG), which uses the verifier-derived advantage as the optimization anchor and validates each privileged correction against the contrast between evidence-present and evidence-degraded teacher views. By admitting only corrections directionally supported by this visual contrast, VAG preserves visually grounded guidance while suppressing inconsistent residuals. Experiments across six visual reasoning benchmarks show that VAG consistently outperforms GRPO and OPSD methods, establishing new state-of-the-art average performance.

Figures & tables

Figure 1: Illustrative contamination diagnostic. (a) A token-level conflict case and (b) the correction-evidence alignment used to motivate attribution.

Figure 2: Comparison of on-policy learning paradigms. (a) RLVR, (b) RLVR with distillation, (c) Visual contrastive distillation, and (d) RLVR with visual attributable distillation.

Figure 3: Overview of the VAG framework. Stage 1: A verifier derives $A_{\mathrm{ORM}}$ from on-policy rollouts. Stage 2: Dual-view teacher evaluation yields the distillation margin $\delta_{t}$ and visual contrast $v_{t}$ ; VAG retains same-sign corrections and discards conflicting ones. Stage 3: The retained correction modulates $A_{\mathrm{ORM}}$ for the clipped GRPO update.

Model	Param.	V*	Zoom	HR-Bench 4K	HR-Bench 8K	MME-RW EN	MME-RW CN	Avg.
Closed-Source Models (Single Forward Pass)
GPT-5.2 ( OpenAI, 2025a )	–	79.06	50.89	81.12	78.38	72.60	68.80	71.81
GPT-5.4 ( OpenAI, 2026 )	–	76.96	52.66	84.00	77.88	74.20	70.93	72.77
Gemini 3.1 Pro ( Google DeepMind, 2026a )	–	87.96	61.18	89.63	86.88	76.53	73.31	79.25
Gemini 3.5 Flash ( Google DeepMind, 2026b )	–	89.01	61.42	89.12	86.62	75.31	73.97	79.24
Open-Source Baselines (Single Forward Pass)

Table 1: Comparison with single-forward-pass MLLMs and post-training methods across multimodal reasoning benchmarks. All entries report accuracy (%).

Table 5

Figure 4: Performance comparison on Qwen3.5-9B.

Figure 5: Prediction and token-importance comparison. VAG produces the correct evidence-grounded prediction and concentrates credit on decision-relevant visual tokens.

Table 8

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Size	V*	Zoom	HR-Bench 4K	HR-Bench 8K	MME-RW EN	MME-RW CN	Avg.
Vision-OPD ( Yuan et al., 2026 )	4B	89.53	59.53	82.25	81.00	74.97	70.71	76.33
OPD-V ( Aniri et al., 2026 )	4B	90.10	61.30	84.88	80.88	75.45	71.49	77.35
VAG (Ours)	4B	91.10	61.89	85.75	82.50	75.96	73.35	78.43
Vision-OPD ( Yuan et al., 2026 )	9B	90.10	64.97	87.12	84.88	73.51	70.44	78.50
OPD-V ( Aniri et al., 2026 )	9B	93.72	63.79	84.38	82.55	71.82	71.05	77.89
VAG (Ours)	9B	92.15	60.12	89.00	87.88	77.68	74.53	80.23

Appendix

Table 6: Evaluation results under the Vision-OPD evaluation protocol.

Benchmark	Samples	Scoring Rule
V* Bench	191	Rule-based answer matching is applied first; unresolved responses are compared with the verified answer by a Yes/No judge.
ZoomBench	845	Generic answer matching handles normalized responses first; unresolved cases use the verified-answer Yes/No judge.
HR-Bench 4K	800	Multiple-choice matching checks direct answer matches and option-letter matches before the Yes/No judge handles unresolved responses.
HR-Bench 8K	800	Multiple-choice matching follows the HR-Bench 4K rule.
MME-RealWorld EN	23609	Multiple-choice matching and the Yes/No judge are applied to the English subset.
MME-RealWorld CN	5917	The Chinese subset uses the MME-RealWorld multiple-choice scoring rule.

Appendix

Table 7: Number of evaluation samples and scoring rule for each benchmark. All methods in Table 1 are evaluated with this identical protocol.

Figure 6: Training prompt and paired visual inputs. The textual prompt, comprising the image placeholder, question, visual focus instruction, answer choices, and output format, is shared across all three passes. The student is conditioned on the original image, the positive teacher on the evidence-centered crop given by the red bounding box, and the negative teacher on a randomly masked version of that crop. The ground-truth answer is used only by the verifier.

Figure 7: Training dynamics of VAG. Left: the mean verifier reward increases over training. Right: the mean response length fluctuates within a broadly constant band and shows no sustained growth. Both panels report raw per-step training statistics without smoothing.

Figure 8: Fine-grained attribute recognition and spatial localization. Top: VAG reads the small helmet region as orange-red and selects the closest option, red, matching the ground truth, whereas OPD-V reports yellow. Bottom: VAG correctly localizes the green region with two oval patterns in the bottom-left corner. OPD-V answers incorrectly in both cases.

Figure 9: Object-part recognition. VAG rules out the parts that are not visible in the frame and identifies the part on the right side of the car as the taillight (option C), matching the ground truth, whereas OPD-V selects an incorrect option.

Figure 10: Chart-based visual arithmetic. The x-axis is labeled only every fifth item. VAG detects that its initial reading yields a value absent from the answer choices, re-derives the bar-to-item mapping, reads Item 10 and Item 15 as $1$ and $11$ , and returns their average $6$ (option D). OPD-V uses unsupported bar values and returns $9$ (option C).

RL for VLMs

Reinforcement learning for vision-language models aligns multimodal reasoning, tool use, and action with task-specific feedback.

See all 452 papers

On-Policy Visual Evidence Distillation
Sep 29, 2026Shaohang Wei, Feifan Song, Guangyue Peng +9VLM Distillation Visually Grounded Reasoning
Peking University · CUHK · Nanjing University +1
VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning
Oct 10, 2026Jiahe FengVLM Distillation Visually Grounded Reasoning
College of Intelligence and Computing, Tianjin University, Tianjin, China
VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation
Jul 30, 2026Kangning Zhang, Yixing Li, Shuai Shao +9Feature Attribution VLM Distillation
Shanghai Jiao Tong University · Xiaohongshu Inc. · The Chinese University of Hong Kong +1

Sep 29, 2026J/K move · Enter open · S save

On-Policy Visual Evidence Distillation

Shaohang Wei, Feifan Song, Guangyue Peng, Wenhao Yu+8

Peking University · CUHK · Nanjing University+1

VLM Distillation Visually Grounded Reasoning On-Policy Distillation

Open paper AskPDF arXiv

Visual agents solve problems by interleaving reasoning with image operations, and on-policy distillation (OPD) provides guidance from a strong teacher on student-generated interaction trajectories. However, image operations change the evidence available for subsequent reasoning, so local errors in evidence acquisition (Acquire), reading (Read), or answer grounding (Ground) can propagate through the trajectory and lead to incorrect answers. Existing multimodal OPD methods primarily construct or contrast auxiliary views of the original image to strengthen supervision, without explicitly modeling the connections between student actions, resulting observations, and subsequent reasoning. This limits their ability to provide corrections tailored to different failure stages. We introduce Reflection on Visual Evidence (ReVuE), an on-policy distillation method for visual agents. ReVuE compares multiple student-generated trajectories for the same query, summarizes the observed visual evidence, and diagnoses the first failure across the Acquire, Read, and Ground stages. The resulting reflections provide training-time context for the teacher. We group and reweight token-level distillation losses according to how strongly these reflections affect the teacher's predictions. This design translates trajectory-level evidence diagnosis into targeted token-level supervision, guiding students to improve their visual evidence acquisition and reasoning. Across 11 benchmarks spanning the Qwen2.5-VL and InternVL3.5 model families, ReVuE outperforms all evaluated OPD baselines in weighted-average scores for perception, mathematical reasoning, and general tasks. ReVuE also reduces redundancy in reasoning and tool calls while improving tool-call accuracy and task accuracy. Code is available at https://github.com/sylvain-wei/ReVuE

Abstract

RL for VLMs

Explore similar work

On-Policy Visual Evidence Distillation

VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning

VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation