Attention Intervention in VLMs

VLM: Vision-Language Model

Follow this topic Ask about this topic

Latest papers

16 papers4 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

RL Teaches VLMs Where to Attend but Not How to Perceive
Oct 14, 2026Ziqian Zhang, Junjie Liao, Zhining Zhang +1RL for VLMs Visual Evidence Sensitivity in VLMs
National University of Singapore · Beijing Normal University · Peking University +1
Attention Sinks in VLMs Are Borrowed, Not Seen
Oct 11, 2026Dengjia Zhang, Alexander Martin, Pengyu Yan +6Attention Intervention in VLMs VLM Interpretability
Johns Hopkins University · University of Pittsburgh · University at Buffalo +2
SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders
Oct 8, 2026Jeonghyo Song, YoungJoon YooAttention Intervention in VLMs Vision-Language Grounding
Department of Artificial Intelligence, Chung-Ang University
Look Where You Say You're Looking: Self-Grounded Attention for Visual Reasoning
Oct 4, 2026Uri Berger, Gal Chechik, Gal DalalVision-Language Grounding VLM Reasoning
NVIDIA Research · The Hebrew University of Jerusalem · University of Melbourne +1
Recompute or Reuse? Diagnosing and Mitigating Textual Shortcuts in VLM Self-Reflection
Aug 3, 2026Wenxiao Fan, Jingling Fu, Fang Li +9Vision-Language Models VLM Reasoning
School of Computer Science, Beijing Institute of Technology · Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University
See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.
Gaze Heads: How VLMs Look at What They Describe
Jun 12, 2026Rohit Gandikota, David BauImage Captioning Visual Attention
Northeastern University
Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
May 24, 2026Yuanzhi Xu, Qian Gao, Jun Fan +4VLM Robustness Vision-Language Models
Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China +1
Attention Hijacking: Response Manipulation Across Queries in Vision-Language Models
May 17, 2026Zhiqiang Wang, Dongrui Liu, Yan Li +4Adversarial Attacks on VLMs Adversarial Attacks
Hong Kong University of Science and Technology · Shanghai Jiao Tong University · Beihang University
The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation
May 7, 2026Hoin Jung, Xiaoqian WangMultimodal RAG Multimodal Grounding
Elmore Family School of Electrical and Computer Engineering Purdue University West Lafayette, IN 47907
Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking
Apr 11, 2026Jingru Li, Wei Ren, Tianqing ZhuJailbreak Attacks Adversarial Attacks on VLMs
China University of Geosciences, Wuhan · City University of Macau
Counting to Four is still a Chore for VLMs
Apr 11, 2026Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van VoAttention Intervention in VLMs Vision-Language Models
MBZUAI
ACT Now: Preempting LVLM Hallucinations via Adaptive Context Integration
Apr 1, 2026Bei Yan, Yuecong Min, Jie Zhang +2Attention Intervention in VLMs VLM Hallucination Mitigation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM
Mar 5, 2026Li'an Zhong, Ziqiang He, Jibin Zheng +3Attention Intervention in VLMs VLM Hallucination Mitigation
Sun Yat-Sen University · Foshan University · University of British Columbia
Decoding the Pulse of Reasoning VLMs in Multi-Image Understanding Tasks
Mar 4, 2026Chenjun LiVLM Reasoning Multi-Image Reasoning
School of Electrical and Computer Engineering, Cornell University
Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs
Feb 17, 2026Guangtao Lyu, Qi Liu, Chenghao Xu +5Attention Intervention in VLMs VLM Hallucination Mitigation
School of Electronic Engineering, Xidian University, Xi’an, China · College of Computer and Information, Hohai University, Nanjing, China · School of Computer Science and Technology, Xidian University, Xi’an, China +1

Oct 14, 2026J/K move · Enter open · S save

RL Teaches VLMs Where to Attend but Not How to Perceive

Ziqian Zhang, Junjie Liao, Zhining Zhang, Wentao Zhu

National University of Singapore · Beijing Normal University · Peking University+1

RL for VLMs Visual Evidence Sensitivity in VLMs VLM Interpretability Attention Intervention in VLMs

Open paper AskPDF arXiv

Although reinforcement learning (RL) effectively trains Vision-Language Model (VLM) agents, the mechanisms behind these capability gains are poorly understood. Across four diverse agent environments, we investigate whether the task-success gains from multi-turn RL are accompanied by improved grounded visual question answering about the environment. To assess this, we design environment-specific grounded VQA tasks that reveal improvements in grounded visual understanding. Representational probing shows that vision-encoder probe accuracy remains relatively stable, while LLM-layer probe accuracy improves. Attention analysis further reveals that LLM layers pay more attention to visual tokens during RL training. To test whether increased attention to visual tokens contributes to VQA gains, we introduce a lightweight plug-and-play intervention using a fixed attention bias. Without updating model parameters or computing gradients, this intervention partially restores VQA gains on the untrained model. These results reveal better utilization of visual information as a key mechanism behind the gains from RL.