Authors: Haowei Wang, Hao Lin, Qi Zhu, Jiayi Ji, Deyi Ji, Liujuan Cao, Rongrong Ji
Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · Moxin Technology
On-policy distillation (OPD) reveals versatility and convenience during VLM post-training, providing dense supervision for student from teacher. However, we observed an anomaly that the distilled VLM could still answer the questions accurately even when the related visual regions were masked or replaced with noise. Building on this observation, our quantitative and qualitative experiments demonstrate that OPD, which uses the answer as a hint, has failed to teach students to genuinely seek out and rely on visual evidence. Our theoretical analysis suggests that this failure arises from a shortcut enabled by the lack of visual constraints during distillation. To address the issue, we introduce Contrastive Vision On-policy Distillation (CV-OPD), which compares teacher distributions induced by positive and negative crops while conditioning on the same student prefix and augments optimization with a margin-based contrastive constraint, which is modulated with discrepancy gating across token and sample level. Specifically, CV-OPD first introduces a negative teacher unrelated to the answer and designs a Contrastive Divergence Loss (CDL) that constrains the feature distribution of student by contrasting the positive and negative teachers, correcting how the student exploits visual regions. The CV-OPD introduces a Discrepancy Aware Gate (DAG) that uses token-level and sample-level discrepancies to select visually relevant tokens for optimization, directing the student's attention toward visual content. Extensive experiments across a broad range of fine-grained visual benchmarks demonstrate that CV-OPD substantially alleviates shortcut and consistently improves the visual reasoning capabilities of existing VLMs.
Figures & tables
Figure 1: (a) The failure mode of Vision-OPD. The distilled VLM can generate correct answer even given wrong visual inputs. (b) Our CV-OPD shows less error with higher accuracy, which is even competitive while compared to closed-source VLMs.
Figure 2: Diagnostic evaluation of visual shortcut. (a) We evaluate accuracy under both black masking and Gaussian noise, where answer-relevant visual regions are replaced. (b) We further visualize the attention ( red rectangle ) of Vision-OPD to find out whether the correct visual information is utilized.
Figure 3: The pipeline of Contrastive Vision On-policy Distillation. CV-OPD introduces a negative teacher that provides unrelated visual information to fix the reliance of visual contents. Samples with at least one correct rollout are retained for training.
Model
Param.
V ⋆
Zoom
HR-Bench [-1pt]4K
HR-Bench [-1pt]8K
MME-RW [-1pt]EN
MME-RW [-1pt]CN
Avg6
Tool-Using and Region-Aware Models
DeepEyes ( Zheng et al., 2026 )
7B
82.63
46.15
75.50
70.00
63.95
62.41
66.77
SenseNova-MARS ( Chng et al., 2025 )
8B
92.15
47.81
83.13
78.38
67.90
68.90
73.05
Vision-RL 2 ( Shi et al., 2026 )
7B
91.62
59.88
78.12
75.00
62.41
58.73
70.96
Closed-Source Models
Gemini 3 Flash ( Google DeepMind, 2025 )
–
85.39
61.49
87.25
85.00
73.15
71.66
77.32
Table 1: Overall comparison on six fine-grained visual benchmarks. We report the models trained on the same data within the Qwen3.5-Based Models. Values are accuracies (%), and Avg6 is the unweighted mean. Within the Qwen3.5-based block, best results are bold and second-best results are underlined; the cyan-green row denotes CV-OPD.
Method
Evaluation
CDL
DAG
V ⋆
Zoom
HR-Bench [-1pt]4K
HR-Bench [-1pt]8K
MME-RW [-1pt]EN
MME-RW [-1pt]CN
Avg6
✗
✗
89.53
59.41
81.75
80.12
74.51
70.17
75.92
✗
✔
90.05
60.71
82.38
82.63
74.67
70.41
76.81
✔
✗
90.58
60.71
83.00
80.88
75.40
70.71
76.88
✔
✔
91.62
62.01
84.38
83.38
76.99
70.93
78.22
Table 2: Ablation of CDL and DAG in the CV-OPD pipeline.
Model
Param.
V ⋆
Zoom
HR-Bench [-1pt]4K
HR-Bench [-1pt]8K
MME-RW [-1pt]EN
MME-RW [-1pt]CN
Avg6
Base
4B
82.20
48.40
86.38
82.00
63.94
63.80
71.12
All Black
4B
91.62
59.76
81.88
81.62
74.94
70.73
76.76
Random Crop
4B
89.53
62.01
84.38
81.50
75.06
70.73
77.20
Neighbor Crop
4B
90.05
59.76
81.62
81.00
75.40
70.54
76.40
CV-OPD
4B
91.62
62.01
84.38
83.38
76.99
70.93
78.22
Table 3: Ablation of negative region selection strategies
DAG
Threshold
V ⋆
Zoom
HR-Bench [-1pt]4K
HR-Bench [-1pt]8K
MME-RW [-1pt]EN
MME-RW [-1pt]CN
Avg6
✗
–
90.58
60.71
83.00
80.88
75.40
70.71
76.88
✔
rollout
90.05
61.54
83.25
82.13
75.05
70.47
77.08
✔
batch
91.62
62.01
84.38
83.38
76.99
70.93
78.22
Table 4: Ablation of DAG threshold settings.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Role
GPU allocation
8 A100 GPUs
Training worker allocation
Backbone
Qwen3.5-4B
Student and EMA teacher model
Training samples
6,241
Number of visual reasoning examples
Training batch size
96
Number of samples per training batch
Actor update mini-batch size
96
Mini-batch size for actor updates
Rollouts per prompt
8
On-policy responses generated per sample
Appendix
Table 5: Training hyperparameters for CV-OPD.
Figure 4: Qualitative comparison of visual attention and model responses between Vision-OPD and CV-OPD. Vision-OPD relies on the dominant color of the mailbox and predicts red, whereas CV-OPD focuses on the lock itself and identifies its metallic silver color.
Figure 5: Attention visualization and model responses under the black-mask setting. After the answer-relevant region is replaced by black pixels, Vision-OPD still produces the expected answer despite failing to localize the relevant visual evidence, indicating shortcut-based prediction and hallucination. In contrast, CV-OPD explicitly describes the observable scene and its ambiguity, showing stronger reliance on available visual evidence.
Figure 6: CV-OPD response under the black-mask setting. When the umbrella is not clearly visible, CV-OPD explicitly acknowledges the lack of reliable color information and limits its description to the remaining visual evidence, although its forced-choice prediction is incorrect.
Figure 7: CV-OPD response under Gaussian noise. With the glove’s material obscured by visual corruption, CV-OPD recognizes that the available evidence is insufficient for reliable identification, despite producing an incorrect final choice.
GRPO-style reinforcement learning has advanced vision-language reasoning, but its sequence-level rewards provide limited credit assignment for individual reasoning tokens. On-policy distillation (OPD) addresses this issue by providing dense supervision on student-generated trajectories, yet standard OPD assigns similar importance to all teacher-preferred tokens, including template phrases and visually irrelevant continuations. We introduce Visual Counterfactual On-Policy Distillation (VC-OPD), a grounding-aware distillation method that prioritizes tokens whose correctness depends on image-specific evidence. After a teacher-supervised warm start that stabilizes response formats and reduces noisy rollouts, VC-OPD performs on-policy distillation with two teacher evaluations for each student trajectory: one under the original image and one under a counterfactual image with corrupted instance-specific evidence. The original-image teacher distribution determines the distillation target, while the original--counterfactual teacher gap estimates token-level visual dependence and softly reweights the loss. To avoid unstable response-level update scales, VC-OPD further applies mass-preserving normalization to the token weights. This design preserves the dense supervision of OPD while shifting learning capacity toward visually grounded reasoning tokens. Experiments on multimodal reasoning benchmarks show that VC-OPD improves visual reasoning performance and produces more effective token-level distillation.
Jiahe Feng
College of Intelligence and Computing, Tianjin University, Tianjin, China
A central goal of vision-language model (VLM) distillation is to transfer both the teacher's language capabilities and its visual understanding. However, existing methods primarily supervise the student's output, leaving visual understanding implicit. Our analysis reveals that a student can match the teacher's answer without relying on the same visual evidence, raising the question: how can we ensure the student responds to the visual information that actually determines the answer? To this end, we propose Cross-World On-Policy Distillation (CW-OPD), which explicitly supervises the student's response to changes in visual evidence. For each example, CW-OPD constructs two visual worlds that share the question and scene context but differ in answer-critical evidence, yielding different answers. We perform on-policy distillation in both worlds and distill the teacher's cross-world belief transition, encouraging the student to match not only what the teacher predicts but also why its prediction changes with the evidence. A gradient analysis shows that this term is invariant to errors shared by both worlds and supplies a corrective signal invisible to endpoint matching alone. In this way, CW-OPD makes reliance on the relevant visual evidence an explicit distillation target rather than an implicit consequence of output matching. To diagnose whether a model truly grounds its answers in visual evidence, we introduce CWBench, which measures cross-world consistency via Cross-World Pair Accuracy (CWPA). Experiments on Qwen3.5-4B show that CW-OPD outperforms the strongest baseline by 1.2 points on average, and the 4B student exceeds DeepSeek-V4.1 (552B) by 22.4 CWPA points on CWBench. Code is released in https://github.com/baokou-fw2/CWAD.
On-policy knowledge distillation has proven effective for language models, yet its application to vision-language models (VLMs) remains underexplored. We observe that standard on-policy distillation can improve a student's output quality while failing to strengthen its reliance on visual input: on vision-critical tokens, the student's predictions remain largely unchanged whether or not fine-grained visual detail is present, even though the teacher's predictions depend heavily on it.To make this difference observable, we introduce visual advantage (VA), the token-level log-probability difference when the teacher scores a student-generated rollout with versus without access to fine-grained visual detail. VA is concentrated in a small minority of tokens, and these high-VA tokens are the ones that actually carry the visual supervision signal. This motivates a distillation objective that treats them differently from language scaffolding, so their contribution is not diluted by the abundant surrounding language tokens.We propose Visual-Advantage On-Policy Distillation (VA-OPD), which uses VA at two granularities: rollout-level reweighting by trajectory-averaged VA, and token-level KL averaged within high-VA and low-VA groups separately. We train on two math datasets (Geometry3K and ViRL39K) and evaluate on eight benchmarks covering both mathematical reasoning and visual understanding, across three teacher sizes (4B, 8B, and 32B) on the Qwen3-VL family. VA-OPD improves over standard on-policy distillation on every benchmark, with the gain growing monotonically along both the teacher-size and data-scale axes, suggesting that these factors compound consistently.
Ruiqi Liu, Xiaolei Lv, Gengsheng Li +8
Institute of Automation, CAS · School of Advanced Interdisciplinary Sciences, UCAS · 3Hello +1