VLM Distillation

VLM: Vision-Language Model

Vision-language model distillation studies how to transfer image-and-text understanding from a teacher model to a smaller or more efficient student, often for tasks such as visual question answering, retrieval, video reasoning, and robotics. Methods align teacher and student features, logits, attention, or relationships between visual and text representations; others train on teacher-generated rationales or student-generated responses, sometimes using multiple teachers or privileged visual cues. Work also targets practical constraints such as fewer visual tokens, quantization, and deployment on devices with limited memory or compute. A central challenge is deciding which teacher signals are reliable and grounded in evidence the student can access, since copying a teacher too closely can preserve its errors or encourage shortcuts rather than visual reasoning.

Latest papers

126 papers29 papers in the last four weeks, up 142% on the four weeks before. 0.2% of all new papers.

OVM-Occ: Open-Vocabulary Monocular Occupancy Prediction via Geometry-Aware Vision-Language Distillation
Oct 26, 2026Shuyang Yu, Haibo Qin, Shaohan Hu +43D Occupancy Prediction 3D Scene Understanding
Zhejiang University. · AgiBot.
RSKD: Grounding Semantic Consistency in Driving VLMs via Distillation
Oct 26, 2026Chuchu Xie, Yang Zhong, Kangan Qian +3VLM Distillation Knowledge Distillation
School of Vehicle and Mobility, Tsinghua University, Beijing, China · Automotive and Robotics, Xiaomi Corporation, Beijing, China
Recursive CLIP: Vision-Language Distillation with Looped Transformer Blocks
Oct 25, 2026Szymon Mazurek, Alberto Presta, Michał ByraVision-Language Models Recurrent Transformers
Samsung AI Center Warsaw, Poland · AGH University of Krakow, Poland · ACC Cyfronet AGH, Poland +1
Edit, Don't Imitate: Minimal-Edit On-Policy Distillation for Multimodal Reasoning
Oct 21, 2026Wen Ye, Ailin Deng, Zirui Wang +8Reasoning for VLMs Vision-Language Models
Meta Superintelligence Labs · University of Southern California · University of California, Berkeley +2
Reinforcement Learning with Visually Attributable On-Policy Self-Distillation
Oct 19, 2026Shiao Xie, Siyu Chen, Jianwei Lv +2RL for VLMs On-Policy Self-Distillation
Baidu Inc., China
A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help
Oct 18, 2026Shashwat Singh, He He, Tal LinzenVision-Language Alignment Cross-Modal Representation Transfer
New York University
When Evidence Vanishes: Distilling Visual-Evidence Dependency in Medical Vision-Language Models
Oct 16, 2026Qizhen Lan, Yiling Ma, Ruiqi Chen +8Medical VLMs Visual Evidence Sensitivity in VLMs
The University of Texas Health Science Center at Houston · Yale University · University of Michigan +3
COMET: Decoupled Distillation, Routing, and Capacity Control for Task-Agnostic Continual Vision--Language Learning
Oct 16, 2026Seungyoon Woo, Keonhee Park, Gunhee KimContinual Learning for VLMs VLM Distillation
Seoul National University
Skill-Decoupled On-Policy Expert Distillation for Unified Vision-Language Models
Oct 15, 2026Yuqi Xiong, Chunyi Peng, Zhipeng Xu +5VLM Distillation RL for VLMs
School of Computer Science and Engineering, Northeastern University, Shenyang, China · Alibaba Group, Hangzhou, China
Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models
Oct 14, 2026Frank F. Yang, Kefei Yan, Yiren Wang +1VLM Distillation On-Policy Distillation
Georgia Institute of Technology, Atlanta, GA, USA
DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction
Oct 13, 2026Haolan Wei, Jiahui Wang, Junli Zha +4VLM Distillation Efficient VLM Inference
SF Technology Co., Ltd, Shenzhen, China
VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning
Oct 10, 2026Jiahe FengVLM Distillation Visually Grounded Reasoning
College of Intelligence and Computing, Tianjin University, Tianjin, China
CV-OPD: Contrastive Vision On-policy Distillation
Oct 9, 2026Haowei Wang, Hao Lin, Qi Zhu +4VLM Distillation Vision-Language Models
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · Moxin Technology
Redirecting Teacher Supervision: Temporal-Response-Guided Latent Distillation for Video-Language Models
Oct 9, 2026Xinkai Lu, Junli Zha, Haolan Wei +3VLM Distillation Knowledge Distillation
SF Technology
Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models
Oct 8, 2026Hongxing Li, Yixin Li, Dingming Li +53D Spatial Reasoning VLM Reasoning
Zhejiang University
KVE-KD: Key Visual Evidence-Guided Knowledge Distillation for Vision-Language Models
Oct 2, 2026Jianbin Zhang, Xin Sun, Shanwen Wang +2VLM Distillation Cross-Modal Knowledge Distillation
Faculty of Data Science, City University of Macau, SAR Macao, China
GRAFT: Growing Agglomerative Foundation Models via Continual Teacher Distillation
Oct 1, 2026Zhenghao Zhao, Chi Zhang, Qingshuang Chen +1Multi-Teacher Knowledge Distillation Vision Foundation Models
University of Illinois Chicago · Amazon
Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes
Oct 1, 2026Sophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc +2VLM Distillation Multimodal Large Language Models
Sorbonne Universit´e, CNRS, ISIR, F-75005 Paris, France · ILLS, CNRS, Montreal, QC H2S 3H1 · Institut universitaire de France (IUF)
HAWK: Rethinking Multimodal Drafting for Speculative Decoding
Sep 30, 2026Wenhan Yang, Anirudh Rao, Ashwin ChandraEfficient Multimodal Inference VLM Distillation
University of California, Los Angeles · Samsung Research America
Less Data, Better Timing: Student-Curriculum Coupling for VLM On-Policy Distillation in Temporal Video Grounding
Sep 30, 2026Jiacheng Qiu, Yunsoo Kim, Ruichen Xu +3Temporal Video Grounding VLM Distillation
State University of New York at Stony Brook
MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models
Sep 30, 2026Yanshu Li, Jiaqian Li, Canran Xiao +3Cross-Modal Knowledge Distillation Multimodal ICL
Brown University · University of Alabama at Birmingham · Hanyang University
Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Models
Sep 30, 2026Yuanhao Sun, Huawei Ji, Jiaxin Ding +2Counterfactual Evaluation of VLMs VLM Distillation
Shanghai Jiao Tong University, Shanghai, China
RS-OPSD: Reliable Privileged On-Policy-Self-Distillation for Ultra-High-Resolution Remote Sensing VQA
Sep 29, 2026Chengjie Jiang, Yunqi Zhou, Jiafeng Yan +3Remote Sensing Image Understanding Visual Question Answering
Tsinghua University · Zhejiang University · Central University of Finance and Economics +2
Improving OCR Faithfulness via Gated and Attenuated On-Policy Distillation
Sep 29, 2026Baode Wang, Zuming Huang, Kexuan Ren +2VLM Distillation On-Policy Distillation
LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillation
Sep 29, 2026Jaeyun Shin, Hangeol Chang, Jong Chul YeVLM Distillation Multi-Teacher Knowledge Distillation
Korea Advanced Institute of Science and Technology (KAIST)
On-Policy Visual Evidence Distillation
Sep 29, 2026Shaohang Wei, Feifan Song, Guangyue Peng +9VLM Distillation Visually Grounded Reasoning
Peking University · CUHK · Nanjing University +1
Evidence-Aligned Multimodal On-Policy Self-Distillation for Fine-Grained Visual Understanding
Sep 28, 2026Nanxing Hu, Qiwei Yan, Jinchao Zhang +1On-Policy Self-Distillation VLM Distillation
Beihang University · University of the Chinese Academy of Sciences
Counterfactual Attention Policy Distillation for Temporal Video Grounding
Sep 28, 2026Shaobo Ju, Haiyang Yu, Xuecheng Wu +5Temporal Video Grounding VLM Distillation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Fudan University. · MMLab, The Chinese University of Hong Kong. +1
Distilling Visual Reasoning into Text Space
Sep 28, 2026Wenhan Yang, Nilay Naharas, Ali Payani +1CoT Distillation VLM Distillation
University of California, Los Angeles · Cisco Research
K-OPSD: Verifiable On-Policy Self-Distillation for Post-Training Vision-Language Models on AEC Drawings
Sep 28, 2026Yunfei Bai, Enrico Chionna, Akash Amol +2On-Policy Self-Distillation VLM Distillation
Amazon Web Services

Oct 26, 2026J/K move · Enter open · S save

OVM-Occ: Open-Vocabulary Monocular Occupancy Prediction via Geometry-Aware Vision-Language Distillation

Shuyang Yu, Haibo Qin, Shaohan Hu, Wei Shan+3

Zhejiang University. · AgiBot.

3D Occupancy Prediction 3D Scene Understanding VLM Distillation

Open paper AskPDF arXiv

Monocular occupancy prediction in indoor scenes aims to recover a scene's 3D geometry and semantics from a single RGB image. Existing methods are restricted to a closed set of predefined categories, limiting their ability to generalize to unseen categories. This limitation is particularly critical in indoor scenes, where objects exhibit highly diverse and fine-grained semantic categories. We study open-vocabulary monocular occupancy prediction for indoor scenes and present OVM-Occ, a Gaussian-based occupancy framework with an instance-aware vision-language teacher. Specifically, we adopt Language-Embedded Gaussians as the intermediate scene representation, propose a geometry-aware deformable attention module to refine Gaussian semantic features, and construct instance-level vision-language embeddings to provide reliable semantic supervision. We further introduce geometry-aware decoupled semantic distillation to mitigate the interference between geometry learning and semantic learning. To evaluate open-vocabulary occupancy prediction performance, we introduce OpenOcc-ScanNet and OpenOcc-Replica, two extended benchmarks with more diverse and fine-grained semantic categories. Experiments show that our method achieves state-of-the-art performance in open-vocabulary monocular occupancy prediction for indoor scenes, outperforming the previous state-of-the-art approach by 12.2 mIoU and 0.6 IoU on the Occ-ScanNet dataset, with consistent gains under new open-vocabulary evaluation protocols.