Attention Intervention in VLMs

VLM: Vision-Language Model

Follow this topic Ask about this topic

Latest papers

15 papers3 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Attention Sinks in VLMs Are Borrowed, Not Seen
Oct 11, 2026Dengjia Zhang, Alexander Martin, Pengyu Yan +6Attention Intervention in VLMs VLM Interpretability
Johns Hopkins University · University of Pittsburgh · University at Buffalo +2
SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders
Oct 8, 2026Jeonghyo Song, YoungJoon YooAttention Intervention in VLMs Vision-Language Grounding
Department of Artificial Intelligence, Chung-Ang University
Look Where You Say You're Looking: Self-Grounded Attention for Visual Reasoning
Oct 4, 2026Uri Berger, Gal Chechik, Gal DalalVision-Language Grounding VLM Reasoning
NVIDIA Research · The Hebrew University of Jerusalem · University of Melbourne +1
Recompute or Reuse? Diagnosing and Mitigating Textual Shortcuts in VLM Self-Reflection
Aug 3, 2026Wenxiao Fan, Jingling Fu, Fang Li +9Vision-Language Models VLM Reasoning
School of Computer Science, Beijing Institute of Technology · Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University
See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.
Gaze Heads: How VLMs Look at What They Describe
Jun 12, 2026Rohit Gandikota, David BauImage Captioning Visual Attention
Northeastern University
Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
May 24, 2026Yuanzhi Xu, Qian Gao, Jun Fan +4VLM Robustness Vision-Language Models
Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China +1
Attention Hijacking: Response Manipulation Across Queries in Vision-Language Models
May 17, 2026Zhiqiang Wang, Dongrui Liu, Yan Li +4Adversarial Attacks on VLMs Adversarial Attacks
Hong Kong University of Science and Technology · Shanghai Jiao Tong University · Beihang University
The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation
May 7, 2026Hoin Jung, Xiaoqian WangMultimodal RAG Multimodal Grounding
Elmore Family School of Electrical and Computer Engineering Purdue University West Lafayette, IN 47907
Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking
Apr 11, 2026Jingru Li, Wei Ren, Tianqing ZhuJailbreak Attacks Adversarial Attacks on VLMs
China University of Geosciences, Wuhan · City University of Macau
Counting to Four is still a Chore for VLMs
Apr 11, 2026Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van VoAttention Intervention in VLMs Vision-Language Models
MBZUAI
ACT Now: Preempting LVLM Hallucinations via Adaptive Context Integration
Apr 1, 2026Bei Yan, Yuecong Min, Jie Zhang +2Attention Intervention in VLMs VLM Hallucination Mitigation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM
Mar 5, 2026Li'an Zhong, Ziqiang He, Jibin Zheng +3Attention Intervention in VLMs VLM Hallucination Mitigation
Sun Yat-Sen University · Foshan University · University of British Columbia
Decoding the Pulse of Reasoning VLMs in Multi-Image Understanding Tasks
Mar 4, 2026Chenjun LiVLM Reasoning Multi-Image Reasoning
School of Electrical and Computer Engineering, Cornell University
Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs
Feb 17, 2026Guangtao Lyu, Qi Liu, Chenghao Xu +5Attention Intervention in VLMs VLM Hallucination Mitigation
School of Electronic Engineering, Xidian University, Xi’an, China · College of Computer and Information, Hohai University, Nanjing, China · School of Computer Science and Technology, Xidian University, Xi’an, China +1

Oct 11, 2026J/K move · Enter open · S save

Attention Sinks in VLMs Are Borrowed, Not Seen

Dengjia Zhang, Alexander Martin, Pengyu Yan, Xihan Sun+5

Johns Hopkins University · University of Pittsburgh · University at Buffalo+2

Attention Intervention in VLMs VLM Interpretability Faithfulness in VLMs

Open paper AskPDF arXiv

We study how to detect and mitigate visual attention sinks in vision-language models (VLMs). Visual attention sinks are image tokens that absorb a large share of attention over the image regardless of the query or task, and they mislead methods that treat attention maps as a record of where the model ``looked.'' Prior work has detected sinks through massive activation dimensions and proposed redistributing their attention, but none have addressed what makes a token become a sink. Comparing six candidate detectors on ten VLMs from four families, we find that only the centered cosine similarity between an image token's key and the key of the language model's text-sink detects visual sinks. Thus, a visual sink is an image token whose key aligns with the text-sink's key. Moving a sink's key component along this direction to the median of the image keys sends its attention back to the text-sink (Are Borrowed) while leaving model answers unchanged~(Not Seen). Building on this, we propose a training-free, entropy-triggered re-look that first cleans the attention map, then crops the most-attended region of the cleaned map and inserts it into the reasoning trace. Cleaning adds negligible computation to the forward pass, and the re-look improves accuracy with a mean gain of

+4.3

points on fine-detail VQA benchmarks across model families.