cs.CVNov 17, 2026

Beyond the Sink Token: Sink-Guided Key Editing in Vision-Language Models

Authors: Hui Zhu, Saleh Zare Zade, Rafi Ibn Sultan, Xiangyu Zhou, Dongxiao Zhu

Organizations: Wayne State University

Abstract

In vision-language models (VLMs), a few sink tokens attract a disproportionate share of attention. While previous studies have examined why these tokens attract attention, we investigate how the sink-key direction affects attention to image and question tokens. Across four probed VLMs, boosting the magnitude of positive sink-aligned components increases attention on average, whereas boosting negative components decreases it; these responses persist through later layers. This identifies the native sink key as a reference for controlling attention beyond its own position. Motivated by this finding, we propose SHRINK, a training-free method that partially reduces sink-aligned components across visible keys during generation. On nine benchmarks under sampled reasoning, it improves mean accuracy by 5.1 and 2.7 percentage points on LLaVA-OneVision-7B and Qwen2.5-VL-3B, with smaller gains on the other two main models, Qwen2.5-VL-7B and Gemma-3-4B. Beyond SHRINK, rescaling sink-aligned key components can replace the attention operations of two existing attention-steering methods, GIFT and ACG, with comparable accuracy on hallucination benchmarks.

Explore similar work

CardsList
  1. Attention Sinks in VLMs Are Borrowed, Not Seen

    Oct 11, 2026Dengjia Zhang, Alexander Martin, Pengyu Yan +6Attention Intervention in VLMsVLM Interpretability

  2. When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

    Apr 1, 2026Jiho Choi, Jaemin Kim, Sanghwan Kim +2Large Vision-Language ModelsVLM Interpretability

  3. Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis

    Sep 1, 2026Minsik Choi, Geewook Kim, Young Geun KimVision-Language ModelsVLM Adaptation