Attention Intervention in VLMs

VLM: Vision-Language Model

Latest papers

28 papers10 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

T3R: Training-Free Two-Stage Token Refinement Towards Efficient and Robust VLA Models
Nov 5, 2026Mohammed Hassan, Zhenyang Chen, Zheng Wang +5Efficient VLA Model Inference Visual Token Pruning
Celine · Georgia Institute of Technology, Atlanta, Georgia, United States · Hong Kong University of Science and Technology, Hong Kong, Hong Kong
$A^3$ -VLA: Automatic Perception-Guided Attention Alignment for Robot Manipulation
Nov 4, 2026Shengzhe Zhang, Qi Zhang, Dazhong Shen +3Vision-Language-Action Models Language-Conditioned Robot Manipulation
School of Computer Science and Technology, University of Science and Technology of China · Shanghai AI Laboratory · College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics +4
Heads That Write, Not Just Point: Image Retrieval Heads in Vision-Language Models
Nov 3, 2026Junsung Park, Sangwon Yu, Donghun Kang +6Large Vision-Language Models VLM Interpretability
Department of Electrical and Computer Engineering, Seoul National University · Upstage AI · Interdisciplinary Program in AI, Seoul National University +3
NAS-PO: Native Attention-Strategy Policy Optimization for Vision-Language Models
Nov 1, 2026Ke An, Shizhe Xiang, Yue Liu +5RL for VLMs Attention Intervention in VLMs
Beijing Institute of Technology · Tianjin University · University of Chinese Academy of Sciences +2
GatedVLA with Camera-Based Emergency Braking: AlpaSim E2E Closed Loop Challenge Technical Report
Nov 1, 2026Kaneyoshi Hiratsuka, Naoki Takada, Makoto KawanoClosed-Loop Control of VLA Models VLMs for Autonomous Driving
Label-Free Localization and Test-Time Suppression of CLIP's Typographic Circuits
Nov 1, 2026Bohan Liu, Wenqian Ye, Guangzhi Xiong +3VLM Robustness Adversarial Attacks on VLMs
University of Virginia Charlottesville VA 22903, USA
FOCUS: Closed-Loop Attention Feedback for Efficient Vision-Language Understanding
Nov 1, 2026Akul Santhosh, Aastha JhunjhunwalaEfficient VLM Inference Video-Language Models
NVIDIA
Reversion, Retention, and Semantic Intervention: Measuring Functional Content in CLIP Fine-Tuning Deltas
Oct 25, 2026Chih-Ting Liao, Kezhen Bao, Xin CaoContrastive Language-Image Pretraining VLM Interpretability
University of New South Wales
Refining Open-Vocabulary Semantic Segmentation via Regional Semantics and Visual Prototypes
Oct 22, 2026Aijing Yu, Zhengbo Wang, Lijun Sheng +2Open-Vocabulary Semantic Segmentation Vision-Language Alignment
School of Cyber Security, University of Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences +2
VLMProbe: An Open-source Library for VLM Probing and Interpretation
Oct 21, 2026Gexin Huang, Haisheng Fu, Yanting Yang +4VLM Interpretability Sparse Autoencoders
The University of British Columbia · Vector Institute
RL Teaches VLMs Where to Attend but Not How to Perceive
Oct 14, 2026Ziqian Zhang, Junjie Liao, Zhining Zhang +1RL for VLMs Visual Evidence Sensitivity in VLMs
National University of Singapore · Beijing Normal University · Peking University +1
Attention Sinks in VLMs Are Borrowed, Not Seen
Oct 11, 2026Dengjia Zhang, Alexander Martin, Pengyu Yan +6Attention Intervention in VLMs VLM Interpretability
Johns Hopkins University · University of Pittsburgh · University at Buffalo +2
SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders
Oct 8, 2026Jeonghyo Song, YoungJoon YooAttention Intervention in VLMs Vision-Language Grounding
Department of Artificial Intelligence, Chung-Ang University
Look Where You Say You're Looking: Self-Grounded Attention for Visual Reasoning
Oct 4, 2026Uri Berger, Gal Chechik, Gal DalalVision-Language Grounding VLM Reasoning
NVIDIA Research · The Hebrew University of Jerusalem · University of Melbourne +1
Recompute or Reuse? Diagnosing and Mitigating Textual Shortcuts in VLM Self-Reflection
Aug 3, 2026Wenxiao Fan, Jingling Fu, Fang Li +9Vision-Language Models VLM Reasoning
School of Computer Science, Beijing Institute of Technology · Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University
See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.
Gaze Heads: How VLMs Look at What They Describe
Jun 12, 2026Rohit Gandikota, David BauImage Captioning Visual Attention
Northeastern University
Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
May 24, 2026Yuanzhi Xu, Qian Gao, Jun Fan +4VLM Robustness Vision-Language Models
Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China +1
Attention Hijacking: Response Manipulation Across Queries in Vision-Language Models
May 17, 2026Zhiqiang Wang, Dongrui Liu, Yan Li +4Adversarial Attacks on VLMs Adversarial Attacks
Hong Kong University of Science and Technology · Shanghai Jiao Tong University · Beihang University
The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation
May 7, 2026Hoin Jung, Xiaoqian WangMultimodal RAG Multimodal Grounding
Elmore Family School of Electrical and Computer Engineering Purdue University West Lafayette, IN 47907
Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking
Apr 11, 2026Jingru Li, Wei Ren, Tianqing ZhuJailbreak Attacks Adversarial Attacks on VLMs
China University of Geosciences, Wuhan · City University of Macau
Counting to Four is still a Chore for VLMs
Apr 11, 2026Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van VoCounting in VLMs Attention Intervention in VLMs
MBZUAI
ACT Now: Preempting LVLM Hallucinations via Adaptive Context Integration
Apr 1, 2026Bei Yan, Yuecong Min, Jie Zhang +2Attention Intervention in VLMs VLM Hallucination Mitigation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM
Mar 5, 2026Li'an Zhong, Ziqiang He, Jibin Zheng +3Attention Intervention in VLMs VLM Hallucination Mitigation
Sun Yat-Sen University · Foshan University · University of British Columbia
Decoding the Pulse of Reasoning VLMs in Multi-Image Understanding Tasks
Mar 4, 2026Chenjun LiVLM Reasoning Multi-Image Reasoning
School of Electrical and Computer Engineering, Cornell University
Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs
Feb 17, 2026Guangtao Lyu, Qi Liu, Chenghao Xu +5Attention Intervention in VLMs VLM Hallucination Mitigation
School of Electronic Engineering, Xidian University, Xi’an, China · College of Computer and Information, Hohai University, Nanjing, China · School of Computer Science and Technology, Xidian University, Xi’an, China +1
Do MLLMs Really See It: Reinforcing Visual Attention in Multimodal LLMs
Feb 9, 2026Siqu Ou, Tianrui Wan, Zhiyuan Zhao +2Reasoning for VLMs Attention Intervention in VLMs
TeleAI · Shanghai Jiao Tong University · Northwestern Polytechnical University
Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs
Jan 20, 2026Yujin Jo, Sangyoon Bae, Taesup KimVLM Hallucination Mitigation Object Hallucination in VLMs
Seoul National University

Nov 5, 2026J/K move · Enter open · S save

T3R: Training-Free Two-Stage Token Refinement Towards Efficient and Robust VLA Models

Mohammed Hassan, Zhenyang Chen, Zheng Wang, Zhixin Zhu+4

Celine · Georgia Institute of Technology, Atlanta, Georgia, United States · Hong Kong University of Science and Technology, Hong Kong, Hong Kong

Efficient VLA Model Inference Visual Token Pruning Robustness of VLA Models Attention Intervention in VLMs

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models enable end-to-end robot control, but inference is increasingly expensive. Each decision processes hundreds of visual tokens, often across multiple cameras, leading to substantial computation while allowing irrelevant background content to distract action prediction under distribution shift. We argue that these issues stem from two complementary forms of redundancy: many visual tokens are spatially irrelevant, while attention over the remaining tokens is often misallocated. Based on this observation, we propose T3R, a Training-Free Two-Stage Token Refinement framework. In the first stage, segmentation-guided token pruning removes irrelevant visual patches to reduce computation and eliminate distracting context. In the second stage, saliency-guided attention biasing reallocates attention toward action-critical visual regions and language instructions within the retained tokens. Together, pruning produces a cleaner candidate set, while attention biasing further emphasizes the most decision-relevant evidence, yielding a better accuracy-efficiency frontier than either stage alone. On LIBERO-Spatial, T3R removes 85% of scene-camera visual tokens while essentially matching the unpruned success rate, and improves success by up to 7 percentage points under out-of-distribution object-placement perturbations. The timing comparison attributes 7.8 ms per inference call to pruning and attention-bias application. As input resolution increases, LLM-forward profiling shows up to a 3.3x speedup over the unpruned baseline. Evaluations across LIBERO, SIMPLER, and RoboTwin with three VLA backbones show the strongest accuracy-efficiency benefits in multi-camera settings. These results demonstrate that training-free token refinement is a practical approach to efficient and robust VLA inference.