cs.CVNov 18, 2026

Keep the Reasoning, Read the Image: On-Demand Visual KV for Multi-Turn Reasoning VLMs

Authors: Ziyi Wang, Xiyue Zhang, Bing Hu, Guangmin Song, Chenyi Zhang, Li Wang

Organizations: Zhejiang University · Zhejiang Sci-Tech University · University of Canterbury · Wenzhou Business College

Abstract

Reasoning vision-language models (VLMs) think for thousands of tokens before they answer, and retention policies keep the KV cache affordable by pruning it as the reasoning grows. We find that this pruning sacrifices the image in multi-turn conversations: the image and the reasoning share one budget, so the growing reasoning crowds the visual entries out. A chart question that loses 8 points to the full cache when asked first loses 24 points when asked after a round of reasoning. Protecting a small visual reserve from eviction recovers most of this loss, while adding the same slots to the shared budget recovers considerably less. The image therefore lacks protection, not capacity. Based on this finding, we propose ShelfKV, a training-free method that works on top of existing retention policies. It keeps every visual entry in CPU memory and, at each decoding step, swaps the visual entries that the current query scores highest into the attention slots of the text entries it scores lowest. The image is never evicted, yet it occupies attention slots only while it is read. We evaluate ShelfKV on four benchmarks, three of them conversational, with two retention policies and two reasoning VLMs. In conversations, it improves the strongest existing policy, DBTrimKV, by up to 20.3 points at matched GPU memory, and still leads by 8.7 points when DBTrimKV uses twice the memory. It also consistently outperforms a fixed visual reserve of matched GPU memory in these settings, without hurting accuracy on standalone questions. These results show that the image should be protected from the reasoning but read only on demand.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. TGV-KV: Text-Grounded KV Eviction for Vision-Language Models

    Jun 2, 2026Jizhihui Liu, Ruizi Han, Miao Zhang +4Vision-Language ModelsEfficient VLM Inference

  2. TraceKV: Policy-Preserving Low-Rank KV Cache Compression for Reasoning Vision-Language Models

    Nov 11, 2026Ke Li, Zishan Shao, Yiran ChenKV-Cache CompressionEfficient VLM Inference

  3. Read Together, Answer Apart: How Earlier Questions Help Vision-Language Models Look

    Nov 18, 2026Xinyi Guo, Kaile Huang, Xitong Ling +5Visual Evidence Sensitivity in VLMsAttention Intervention in VLMs