Let a Small Model Read First: Faster Prefill for Long Interleaved Image-Text Inputs
Organizations: Technische Universität Darmstadt · National Yang Ming Chiao Tung University · TU Ilmenau
Abstract
Vision-language models (VLMs) can answer questions about long interleaved inputs, such as documents of many pages and collections of hundreds of images, but they are slow to start: on inputs of about 100K tokens, the prefill takes most of the wait for the first output token. A common approach prunes the input by scoring its tokens with the target model itself or with a small model of the same family that first writes an answer. However, this scoring takes much of the time that pruning saves. We find that the evidence for a question lies in a few whole units (images or paragraphs), and that a small hybrid model finds them with one attention head of an early layer, even at half image resolution. Based on these findings, we propose SkimPrefill, which lets the small hybrid model read the whole input first, up to that early layer, with images at half resolution. Most layers of the small hybrid model (Qwen3.5-0.8B) run in linear time, so this pass is short. SkimPrefill then ranks every unit with that head and gives the target the top units as an ordinary shorter prompt. One untrained selector serves targets of different model families. Extensive experiments on MMLongBench at 128K tokens, with 19 datasets of five task types, show the effectiveness of SkimPrefill. With 10% of the input tokens kept, SkimPrefill speeds up the prefill of Qwen3.5-9B, Gemma-4-12B and Qwen3-VL-32B by 3.7 times, 11.6 time and 8.2 times while maintaining their average accuracy.
Figures & tables
| Method | Training-free | VLM | Other families | Kept unit | Scorer |
|---|---|---|---|---|---|
| FastV [ 5 ] | ✓ | ✓ | ✗ | image tokens | target, early layer |
| FlexSelect [ 59 ] (training-free form) | ✓ | ✓ | ✗ | video tokens | target, middle layer |
| SGP [ 60 ] | ✓ | ✓ | ✗ | image tokens | small VLM, all layers |
| IF-Prune [ 37 ] | ✗ | ✓ | ✗ | image tokens | trained small VLM |
| LongLLMLingua [ 17 ] | ✓ | ✗ | ✓ | documents, tokens | small LM |
| SpecPrefill [ 25 ] | ✓ | ✗ | ✗ | text blocks | small LLM, all layers |
| Accuracy on MMLongBench, 128K | Prefill | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Target | Method | Selection by | VRAG | NIAH | ICL | Summ | DocQA | Avg. | Kept | Speedup |
| Full input | – | 62.0 | 60.5 | 85.5 | 22.5 | 72.1 | 60.5 | 100% | 0 1.0 | |
| FastV [ 5 ] | target, one layer | 61.0 | 42.0 | 50.8 | 18.4 | 21.0 | 38.6 | 0 10% | 0 1.6 | |
| FlexSelect [ 59 ] | target, reference layer | 62.0 | 64.7 | 85.5 | 20.6 | 75.0 | 61.6 | 0 10% | 0 0.9 | |
| SGP [ 60 ] | small model, same family | 60.5 | 59.7 | 83.8 | 21.0 | 73.2 | 59.6 | 0 10% | 0 1.1 | |
| SpecPrefill [ 25 ] | small model, 32-token chunks | 62.5 | 61.1 | 58.8 | 20.0 | 63.4 | 53.1 | 0 10% | 0 1.0 | |
| Qwen3.5-9B | Gemma-4-12B | Qwen3-VL-32B | ||||
|---|---|---|---|---|---|---|
| Method | Tokens | 32K | Tokens | 32K | Tokens | 32K |
| Full input | 6.4K | 102 | 10.4K | 435 | 1.9K | 50 |
| FastV | 9.1K | 204 | 18.9K | 875 | 5.4K | 218 |
| FlexSelect | 5.9K | 105 | 22.5K | 1,007 | 4.1K | 169 |
| SGP | 6.3K | 115 | 14.3K | 626 | 3.9K | 158 |
| SpecPrefill | 6.5K | 106 | 24.0K | 1,175 | 2.9K | 98 |
| Qwen3.5-9B | Gemma-4-12B | |||
| Variant | Acc. | Speedup | Acc. | Speedup |
| Full input | 60.5 | 1.0 | 35.0 | 1.0 |
| SkimPrefill | 62.7 | 3.7 | 45.4 | 11.6 |
| What is kept | ||||
| Top tokens, original positions | 37.9 | – | 15.4 | – |
| Top tokens, consecutive positions | 38.8 | – | 25.5 | – |
| Accuracy | ||||
| Method | Pages | Missed | Qwen3-VL-32B | Gemma-4-12B |
| Full input | 57.0 | 0 | 76.2 | 57.8 |
| SGP, 10% | 73.9 | 24.3 | ||
| SkimPrefill, 5% | 2.6 | 39 | 45.8 | 40.2 |
| SkimPrefill, 10% | 6.1 | 18 | 62.5 | 53.0 |
| SkimPrefill, 15% | 9.5 | 12 | 66.0 | 56.1 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Layer | Head | Layers run | Image | Text |
|---|---|---|---|---|
| 3 | 7 | 4 | 49 | 25 |
| 7 | 7 | 8 | 50 | 31 |
| 11 | 3 | 12 | 47 | 38 |
| 15 | 2 | 16 | 46 | 34 |
| 19 | 5 | 20 | 47 | 31 |
| 23 | 3 | 24 | 47 | 27 |
| Qwen3.5-9B | Gemma-4-12B | Qwen3-VL-32B | ||||||||||||||||
| Dataset | Full | FastV | Flex. | SGP | Spec. | Ours | Full | FastV | Flex. | SGP | Spec. | Ours | Full | FastV | Flex. | SGP | Spec. | Ours |
| Visual RAG | ||||||||||||||||||
| InfoSeek | 53.0 | 40.0 | 50.0 | 52.0 | 51.0 | 52.0 | 40.0 | 5.0 | 12.0 | 25.0 | 19.0 | 43.0 | 50.0 | 35.0 | 53.0 | 45.0 | 42.0 | 54.0 |
| ViQuAE | 71.0 | 82.0 | 74.0 | 69.0 | 74.0 | 76.0 | 60.0 | 4.0 | 24.0 | 69.0 | 39.0 | 66.0 | 70.0 | 58.0 | 74.0 | 69.0 | 74.0 | 81.0 |
| Needle in a haystack | ||||||||||||||||||
| NIAH retrieval, image | 60.0 | 44.0 | 80.0 | 74.0 | 70.0 | 88.0 | 14.0 | 14.0 | 28.0 | 18.0 | 24.0 | 68.0 | 72.0 | 26.0 | 78.0 | 90.0 | 42.0 | 78.0 |
| Qwen3.5-9B | Gemma-4-12B | Qwen3-VL-32B | ||||
|---|---|---|---|---|---|---|
| Method | Off | On | Off | On | Off | On |
| Full input | 55.4 | 60.5 | 30.5 | 35.0 | 57.0 | 59.6 |
| FastV [ 5 ] | 35.7 | 38.6 | 0 8.1 | 10.4 | 36.6 | 34.2 |
| FlexSelect [ 59 ] | 56.8 | 61.6 | 10.7 | 12.4 | 58.6 | 59.0 |
| SGP [ 60 ] | 55.9 | 59.6 | 24.6 | 27.9 | 57.5 | 59.3 |
| SpecPrefill [ 25 ] | 55.5 | 53.1 | 18.1 | 20.7 | 55.9 | 53.1 |
| Method | Before | Prefill | Decode | Total | Speedup | Tokens | ms/tok |
|---|---|---|---|---|---|---|---|
| Qwen3.5-9B | |||||||
| Full input | 1.37 | 5.82 | 56.3 | 63.5 | 1.00 | 6,924 | 8.1 |
| FastV | 1.37 | 3.06 | 67.5 | 71.9 | 0.88 | 9,152 | 7.4 |
| FlexSelect | 5.38 | 2.25 | 44.4 | 52.0 | 1.22 | 6,322 | 7.0 |
| SGP | 4.22 | 2.44 | 48.0 | 54.7 | 1.16 | 6,625 | 7.2 |
| SpecPrefill | 5.03 | 1.88 | 58.0 | 65.0 | 0.98 | 8,249 | 7.0 |
| Target | GPUs | Before | Prefill | Total | Speedup |
|---|---|---|---|---|---|
| Qwen3.5-9B | Separate | 1.45 | 0.48 | 1.93 | 3.7 |
| Shared | 1.46 | 0.48 | 1.94 | 3.7 | |
| Gemma-4-12B | Separate | 1.35 | 1.59 | 2.95 | 11.6 |
| Shared | 1.35 | 1.59 | 2.94 | 11.6 | |
| Qwen3-VL-32B | Separate | 1.46 | 0.87 | 2.33 | 8.2 |
| Shared | 1.48 | 0.87 | 2.35 | 8.1 |
| Full input | Selector, full res. | Selector, half res. (ours) | |||
|---|---|---|---|---|---|
| Target | Avg. | Avg. | Speedup | Avg. | Speedup |
| Qwen3.5-9B | 60.5 | 63.4 | 2.0 | 62.7 | 3.7 |
| Gemma-4-12B | 35.0 | 45.5 | 7.1 | 45.4 | 11.6 |
| Qwen3-VL-32B | 59.6 | 62.2 | 4.8 | 62.4 | 8.2 |
| Qwen3.5-9B | Gemma-4-12B | |||||
|---|---|---|---|---|---|---|
| Selector | Layers | Avg. | Speedup | Avg. | Speedup | Select |
| Full input | – | 60.5 | 1.0 | 35.0 | 1.0 | – |
| Qwen3.5-0.8B (ours) | 8 | 62.7 | 3.7 | 45.4 | 11.6 | 0.71 |
| Qwen3.5-2B | 12 | 62.7 | 3.2 | 45.5 | 10.3 | 0.96 |
| Qwen3-VL-2B | 12 | 62.7 | 2.9 | 46.0 | 9.6 | 1.20 |
| Scorer | VRAG | NIAH | ICL | Summ | DocQA | Avg. | Speedup |
|---|---|---|---|---|---|---|---|
| Full input | 62.0 | 60.5 | 85.5 | 22.5 | 72.1 | 60.5 | 1.0 |
| SigLIP 2 | 55.0 | 72.6 | 91.3 | 18.0 | 49.8 | 57.3 | 4.0 |
| ColEmbed V2 8B | 57.5 | 77.0 | 86.8 | 19.8 | 70.9 | 62.4 | 1.3 |
| GME-2B | 61.0 | 65.6 | 90.0 | 18.8 | 70.5 | 61.2 | 1.6 |
| SkimPrefill (ours) | 64.0 | 75.0 | 87.0 | 18.3 | 69.4 | 62.7 | 3.7 |
| Scorer | NIAH, image | NIAH, text | VH | DocQA |
|---|---|---|---|---|
| SigLIP 2 | 96.0 | 88.4 | 41.3 | 54.3 |
| ColEmbed V2 8B | 96.7 | 88.4 | 67.8 | 85.7 |
| GME-2B | 94.0 | 78.4 | 41.8 | 78.3 |
| SkimPrefill (ours) | 96.7 | 92.0 | 71.5 | 79.4 |
| Accuracy | Many-shot ICL | |||||||
| Input | VRAG | NIAH | ICL | Summ | DocQA | Avg. | Tokens | 32K |
| Qwen3-VL-32B | ||||||||
| Full input | 60.0 | 58.5 | 82.8 | 22.3 | 74.6 | 59.6 | 0.7K | 5 |
| 5% | 62.5 | 71.7 | 81.3 | 17.3 | 59.6 | 58.5 | 1.0K | 2 |
| 10% | 67.5 | 69.4 | 84.8 | 18.9 | 71.4 | 62.4 | 1.3K | 4 |
| 15% | 64.0 | 71.2 | 85.8 | 19.8 | 71.9 | 62.5 | 1.1K | 5 |
Efficient VLM Inference
Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.
See all 530 papers