cs.CVNov 19, 2026

Fold the Loop: Decoding-Time KV Memory for Looped Vision-Language Models

Authors: Kun Zhou, Wenmian Yang, Zhensheng Wang, Yanzeng Li, Weijia Jia

Abstract

Looped vision-language models (VLMs) deepen visual reasoning by repeatedly applying weight-tied modules. However, their decoding memory grows with recurrent depth and limits the image size and batch size that fit in a fixed memory budget. Each recurrent step produces its own key-value (KV) states, so the cache holds several copies of the same image. We find that, after an exact prefill, earlier steps still need access to visual information but can reuse the final visual KV within the same module. We propose FoldCache, which compresses visual KV along the recurrent axis while preserving the full prefill computation. FoldCache partitions recurrent steps within each module into groups and retains the final visual KV of each group for its members to share during decoding. Sharing stays within module boundaries, while question and answer KV remain separate for each step. The grouping granularity controls the degree of compression, and folding can be combined with token eviction and quantization to further reduce memory use. On LoopVL, FoldCache approximately halves prefix KV memory while matching full-cache accuracy on five open-ended benchmarks. These memory savings support batch sizes up to 1.7 times those of a full cache under the same GPU memory budget, and up to 2.6 times with the more aggressive FoldCache-Lite.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference

    Aug 25, 2026Lyuke Wang, Zhuo Li, Guangxu ZhuEfficient VLM InferenceLong-Context VLMs

  2. KVCapsule: Efficient Sequential KV Cache Compression for Vision-Language Models with Asymmetric Redundancy

    May 14, 2026Yingbing Huang, Tharun Adithya Srikrishnan, Steven K. Reinhardt +1Vision-Language ModelsEfficient VLM Inference

  3. Make Your LVLM KV Cache More Lightweight

    May 1, 2026Xihao Chen, Yangyang Guo, Roger ZimmermannKV CachingEfficient VLM Inference