Attention-Guided Distillation

Follow this topic Ask about this topic

Latest papers

11 papers1 paper in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Perception or Reasoning? Diagnosing and Tackling Reinforcement Learning Bottlenecks in VLMs
Oct 15, 2026Zeyuan Yang, Zhuohao Chen, Jiaben Chen +3Attention-Guided Distillation RL for VLMs
UMass Amherst · UIUC · MIT
GAD: Geometry Attention Distillation for Consistent Video Generation
Oct 13, 2026Junfeng Ni, Ziyu Zhu, Fan Jiang +7Attention-Guided Distillation Geometric Consistency in Video Generation
Tsinghua University · AMAP, Alibaba Group · State Key Laboratory of General Artificial Intelligence, BIGAI
Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning
Sep 27, 2026Safaeid Hossain Arib, Rabeya Akter, Ismam Nur Swapnil +3Numerical Reasoning in Language Models Attention-Guided Distillation
ACI PLC, Bangladesh · University of Dhaka, Bangladesh · BRAC University, Bangladesh +2
Visual Saliency Steering Distillation for Multimodal Chain-of-Thought Reasoning
Jul 24, 2026Hao Yang, Jin Wang, Xuejie ZhangAttention-Guided Distillation VLM Distillation
School of Information Science and Engineering, Yunnan University, Kunming, China
SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
Jun 22, 2026Huzama Ahmad, Se-Young YunSparse Attention Block-Sparse Attention
KAIST
Denoising-Enhanced Coarse-to-Fine Infrared Small Target Detection with Attention Prior-Guided Knowledge Distillation
Jun 20, 2026Houzhang Fang, Ruixuan Huang, Qiuhuan Chen +3Attention-Guided Distillation Infrared Small Target Detection
Xidian University, Xi’an, China · Huazhong University of Science and Technology, Wuhan, China
StreamKL: Fast and Memory-Efficient KL Divergence for Boosting Attention Distillation
Jun 18, 2026Guangda Liu, Yiquan Wang, Chengwei Li +6GPU Kernel Optimization Attention-Guided Distillation
Shanghai Jiao Tong University · Huawei · Fudan University
Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces
Jun 5, 2026Debjyoti Saha RoyMulti-Label Classification CoT Distillation
Khoury College of Computer Sciences, Northeastern University
Evaluating Explainable AI Attribution Methods in Neural Machine Translation via Attention-Guided Knowledge Distillation
Mar 11, 2026Aria Nourbakhsh, Salima Lamsiyah, Adelaide Danilov +1Attention-Guided Distillation Explainability Evaluation
Department of Computer Science, University of Luxembourg, 4364 Esch-sur-Alzette, Luxembourg
FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models
Mar 9, 2026Haoyang Li, Liang Wang, Siyu Zhou +5Attention-Guided Distillation VLM Adaptation
University of Technology Sydney, NSW 2007, Australia · Shanghai University, Shanghai 200444, China
LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
Mar 9, 2026Ning Yang, Hengyu Zhong, Wentao Wang +3Attention-Guided Distillation Memory-Efficient Attention
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Southwest University, Chongqing, China · Dalian University of Technology, Dalian, Liaoning, China +3

Oct 15, 2026J/K move · Enter open · S save

Perception or Reasoning? Diagnosing and Tackling Reinforcement Learning Bottlenecks in VLMs

Zeyuan Yang, Zhuohao Chen, Jiaben Chen, Yao Zhang+2

UMass Amherst · UIUC · MIT

Attention-Guided Distillation RL for VLMs Vision-Language Models VLM Reasoning Reasoning for VLMs VLM Evaluation+1

Open paper AskPDF arXiv

Reinforcement learning has brought strong gains to language model reasoning, but its improvements on visual tasks remain less consistent. One difficulty is that visual task failures can arise from different sources: the same benchmark may be limited by perception for one model and by reasoning for another. We introduce Perception and Reasoning Attribution (PARA), which measures the remaining perceptual and reasoning headroom of each model--benchmark pair by comparing how many current errors are recovered with more accurate visual information versus a stronger reasoner. Across 4 vision-language models and 17 multimodal benchmarks, PARA broadly agrees with the expected characteristics of existing benchmarks while revealing substantial model-specific variation. More importantly, it shows that RL gains become smaller as perceptual headroom increases, but larger when reasoning explains more of the recoverable errors, suggesting that current RL exploits reasoning headroom more effectively than perceptual headroom. Motivated by this gap, we analyze how existing RL objectives affect visual perception and find that reward- and token-level signals influence visual attention only indirectly. We therefore propose GRAD, a group-relative attention distillation objective that uses the best rollout in each group to directly guide the visual attention of weaker rollouts. GRAD consistently improves performance across two model scales and seven benchmarks, and further complements existing RL methods.