Multimodal reinforcement learning with verifiable rewards (RLVR) provides reliable task-level supervision, yet offers limited guidance for localizing errors in visual reasoning. On-policy self-distillation (OPSD) complements this supervision with dense token-level corrections from a privileged teacher equipped with an evidence-centered visual view. However, the resulting corrections may conflate visual evidence with non-visual shifts induced by linguistic priors, response style, or the privileged view itself. We formalize this phenomenon as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence. Directly distilling contaminated corrections may therefore reinforce erroneous token-level updates. To address this issue, we propose reinforcement learning with visually attributable on-policy self-distillation, an evidence-consistent paradigm that augments task-level RLVR supervision with privileged token-level corrections grounded in visual evidence. We instantiate this paradigm with Visual-Attribution Gating (VAG), which uses the verifier-derived advantage as the optimization anchor and validates each privileged correction against the contrast between evidence-present and evidence-degraded teacher views. By admitting only corrections directionally supported by this visual contrast, VAG preserves visually grounded guidance while suppressing inconsistent residuals. Experiments across six visual reasoning benchmarks show that VAG consistently outperforms GRPO and OPSD methods, establishing new state-of-the-art average performance.
Figures & tables
Figure 1: Illustrative contamination diagnostic. (a) A token-level conflict case and (b) the correction-evidence alignment used to motivate attribution.
Figure 2: Comparison of on-policy learning paradigms. (a) RLVR, (b) RLVR with distillation, (c) Visual contrastive distillation, and (d) RLVR with visual attributable distillation.
Figure 3: Overview of the VAG framework. Stage 1: A verifier derives AORM from on-policy rollouts. Stage 2: Dual-view teacher evaluation yields the distillation margin δt and visual contrast vt ; VAG retains same-sign corrections and discards conflicting ones. Stage 3: The retained correction modulates AORM for the clipped GRPO update.
Model
Param.
V*
Zoom
HR-Bench 4K
HR-Bench 8K
MME-RW EN
MME-RW CN
Avg.
Closed-Source Models (Single Forward Pass)
GPT-5.2 ( OpenAI, 2025a )
–
79.06
50.89
81.12
78.38
72.60
68.80
71.81
GPT-5.4 ( OpenAI, 2026 )
–
76.96
52.66
84.00
77.88
74.20
70.93
72.77
Gemini 3.1 Pro ( Google DeepMind, 2026a )
–
87.96
61.18
89.63
86.88
76.53
73.31
79.25
Gemini 3.5 Flash ( Google DeepMind, 2026b )
–
89.01
61.42
89.12
86.62
75.31
73.97
79.24
Open-Source Baselines (Single Forward Pass)
Table 1: Comparison with single-forward-pass MLLMs and post-training methods across multimodal reasoning benchmarks. All entries report accuracy (%).
Table 5
Figure 4: Performance comparison on Qwen3.5-9B.
Figure 5: Prediction and token-importance comparison. VAG produces the correct evidence-grounded prediction and concentrates credit on decision-relevant visual tokens.
Table 8
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Size
V*
Zoom
HR-Bench 4K
HR-Bench 8K
MME-RW EN
MME-RW CN
Avg.
Vision-OPD ( Yuan et al., 2026 )
4B
89.53
59.53
82.25
81.00
74.97
70.71
76.33
OPD-V ( Aniri et al., 2026 )
4B
90.10
61.30
84.88
80.88
75.45
71.49
77.35
VAG (Ours)
4B
91.10
61.89
85.75
82.50
75.96
73.35
78.43
Vision-OPD ( Yuan et al., 2026 )
9B
90.10
64.97
87.12
84.88
73.51
70.44
78.50
OPD-V ( Aniri et al., 2026 )
9B
93.72
63.79
84.38
82.55
71.82
71.05
77.89
VAG (Ours)
9B
92.15
60.12
89.00
87.88
77.68
74.53
80.23
Appendix
Table 6: Evaluation results under the Vision-OPD evaluation protocol.
Benchmark
Samples
Scoring Rule
V* Bench
191
Rule-based answer matching is applied first; unresolved responses are compared with the verified answer by a Yes/No judge.
ZoomBench
845
Generic answer matching handles normalized responses first; unresolved cases use the verified-answer Yes/No judge.
HR-Bench 4K
800
Multiple-choice matching checks direct answer matches and option-letter matches before the Yes/No judge handles unresolved responses.
HR-Bench 8K
800
Multiple-choice matching follows the HR-Bench 4K rule.
MME-RealWorld EN
23609
Multiple-choice matching and the Yes/No judge are applied to the English subset.
MME-RealWorld CN
5917
The Chinese subset uses the MME-RealWorld multiple-choice scoring rule.
Appendix
Table 7: Number of evaluation samples and scoring rule for each benchmark. All methods in Table 1 are evaluated with this identical protocol.
Figure 6: Training prompt and paired visual inputs. The textual prompt, comprising the image placeholder, question, visual focus instruction, answer choices, and output format, is shared across all three passes. The student is conditioned on the original image, the positive teacher on the evidence-centered crop given by the red bounding box, and the negative teacher on a randomly masked version of that crop. The ground-truth answer is used only by the verifier.
Figure 7: Training dynamics of VAG. Left: the mean verifier reward increases over training. Right: the mean response length fluctuates within a broadly constant band and shows no sustained growth. Both panels report raw per-step training statistics without smoothing.
Figure 8: Fine-grained attribute recognition and spatial localization. Top: VAG reads the small helmet region as orange-red and selects the closest option, red, matching the ground truth, whereas OPD-V reports yellow. Bottom: VAG correctly localizes the green region with two oval patterns in the bottom-left corner. OPD-V answers incorrectly in both cases.
Figure 9: Object-part recognition. VAG rules out the parts that are not visible in the frame and identifies the part on the right side of the car as the taillight (option C), matching the ground truth, whereas OPD-V selects an incorrect option.
Figure 10: Chart-based visual arithmetic. The x-axis is labeled only every fifth item. VAG detects that its initial reading yields a value absent from the answer choices, re-derives the bar-to-item mapping, reads Item 10 and Item 15 as 1 and 11 , and returns their average 6 (option D). OPD-V uses unsupported bar values and returns 9 (option C).
Visual agents solve problems by interleaving reasoning with image operations, and on-policy distillation (OPD) provides guidance from a strong teacher on student-generated interaction trajectories. However, image operations change the evidence available for subsequent reasoning, so local errors in evidence acquisition (Acquire), reading (Read), or answer grounding (Ground) can propagate through the trajectory and lead to incorrect answers. Existing multimodal OPD methods primarily construct or contrast auxiliary views of the original image to strengthen supervision, without explicitly modeling the connections between student actions, resulting observations, and subsequent reasoning. This limits their ability to provide corrections tailored to different failure stages. We introduce Reflection on Visual Evidence (ReVuE), an on-policy distillation method for visual agents. ReVuE compares multiple student-generated trajectories for the same query, summarizes the observed visual evidence, and diagnoses the first failure across the Acquire, Read, and Ground stages. The resulting reflections provide training-time context for the teacher. We group and reweight token-level distillation losses according to how strongly these reflections affect the teacher's predictions. This design translates trajectory-level evidence diagnosis into targeted token-level supervision, guiding students to improve their visual evidence acquisition and reasoning. Across 11 benchmarks spanning the Qwen2.5-VL and InternVL3.5 model families, ReVuE outperforms all evaluated OPD baselines in weighted-average scores for perception, mathematical reasoning, and general tasks. ReVuE also reduces redundancy in reasoning and tool calls while improving tool-call accuracy and task accuracy. Code is available at https://github.com/sylvain-wei/ReVuE
GRPO-style reinforcement learning has advanced vision-language reasoning, but its sequence-level rewards provide limited credit assignment for individual reasoning tokens. On-policy distillation (OPD) addresses this issue by providing dense supervision on student-generated trajectories, yet standard OPD assigns similar importance to all teacher-preferred tokens, including template phrases and visually irrelevant continuations. We introduce Visual Counterfactual On-Policy Distillation (VC-OPD), a grounding-aware distillation method that prioritizes tokens whose correctness depends on image-specific evidence. After a teacher-supervised warm start that stabilizes response formats and reduces noisy rollouts, VC-OPD performs on-policy distillation with two teacher evaluations for each student trajectory: one under the original image and one under a counterfactual image with corrupted instance-specific evidence. The original-image teacher distribution determines the distillation target, while the original--counterfactual teacher gap estimates token-level visual dependence and softly reweights the loss. To avoid unstable response-level update scales, VC-OPD further applies mass-preserving normalization to the token weights. This design preserves the dense supervision of OPD while shifting learning capacity toward visually grounded reasoning tokens. Experiments on multimodal reasoning benchmarks show that VC-OPD improves visual reasoning performance and produces more effective token-level distillation.
Jiahe Feng
College of Intelligence and Computing, Tianjin University, Tianjin, China
Multimodal on-policy distillation (OPD) transfers fine-grained visual knowledge by supervising student-generated trajectories with a privileged-view teacher. Yet its next-token corrections are source-mixed, combining visual signals with linguistic priors and teacher-specific effects. The key challenge is to estimate which corrections are supported by visual evidence, not merely where or how strongly to distill. We introduce Visual Attribution Distillation (VAD), a counterfactual target-reconstruction algorithm that estimates the visually attributable part of a teacher correction. At each student-generated prefix, VAD evaluates the same fixed teacher with the relevant evidence present and removed. The corresponding change in centered log-probabilities defines ut, a signed proxy for the visual evidence direction that estimates how revealing the evidence supports or refutes candidate tokens. VAD projects the original correction onto this proxy to obtain an intervention-aligned component and a proxy-unexplained residual, then reconstructs a student-anchored target from the former. During training, this reconstructed target supplies the primary supervision signal, while the privileged teacher contributes a weak regularizer. Across six fine-grained visual benchmarks at 4B and 9B scales, VAD outperforms direct privileged-view distillation and visual-advantage weighting. Token- level and controlled-target analyses show that the proxy-aligned component is enriched in task-relevant visual corrections and yields stronger target shifts, especially when evidence refutes a mistaken answer. These results support counterfactual target reconstruction as an effective alternative to source-mixed supervision.
Kangning Zhang, Yixing Li, Shuai Shao +9
Shanghai Jiao Tong University · Xiaohongshu Inc. · The Chinese University of Hong Kong +1