cs.CVNov 18, 2026

Let a Small Model Read First: Faster Prefill for Long Interleaved Image-Text Inputs

Authors: Xiaolin Lin, Kai-Chiang Wu, Bing Li, Grace Li Zhang

Organizations: Technische Universität Darmstadt · National Yang Ming Chiao Tung University · TU Ilmenau

Abstract

Vision-language models (VLMs) can answer questions about long interleaved inputs, such as documents of many pages and collections of hundreds of images, but they are slow to start: on inputs of about 100K tokens, the prefill takes most of the wait for the first output token. A common approach prunes the input by scoring its tokens with the target model itself or with a small model of the same family that first writes an answer. However, this scoring takes much of the time that pruning saves. We find that the evidence for a question lies in a few whole units (images or paragraphs), and that a small hybrid model finds them with one attention head of an early layer, even at half image resolution. Based on these findings, we propose SkimPrefill, which lets the small hybrid model read the whole input first, up to that early layer, with images at half resolution. Most layers of the small hybrid model (Qwen3.5-0.8B) run in linear time, so this pass is short. SkimPrefill then ranks every unit with that head and gives the target the top units as an ordinary shorter prompt. One untrained selector serves targets of different model families. Extensive experiments on MMLongBench at 128K tokens, with 19 datasets of five task types, show the effectiveness of SkimPrefill. With 10% of the input tokens kept, SkimPrefill speeds up the prefill of Qwen3.5-9B, Gemma-4-12B and Qwen3-VL-32B by 3.7 times, 11.6 time and 8.2 times while maintaining their average accuracy.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Read Together, Answer Apart: How Earlier Questions Help Vision-Language Models Look

    Nov 18, 2026Xinyi Guo, Kaile Huang, Xitong Ling +5Visual Evidence Sensitivity in VLMsAttention Intervention in VLMs

  2. ReToken: Improving Long-Context VLMs with Visual Retrieval Token

    Jul 30, 2026Yao Xiao, Reuben Tan, Zhen Zhu +3Vision-Language ModelsVision-Language Retrieval

  3. SEE: Unlocking the Hidden Perception of Frozen VLMs

    Nov 18, 2026Alin-Ionut Popa, Tuo Zhang, Dimitrios DimitriadisVLM InferenceVisual Evidence Sensitivity in VLMs