SIEVE: Preserving Cross-Image Evidence via Depth-Aware Token Compression
Organizations: Michigan State University · DatologyAI · Carnegie Mellon University
Abstract
Multi-image vision-language models (VLMs) incur substantial computation when processing long visual sequences. Compressing these sequences must preserve complementary evidence across images and keep tokens from different images separate. Global selection can underrepresent question-relevant images, and pruning in early decoder layers risks removing relevant content before sufficient multimodal processing, yet existing methods spend most of the budget on these early layers. We propose SIEVE, a training-free, single-pass framework that jointly allocates visual tokens across images and decoder depth. SIEVE first pools each image into a compact spatial grid to reduce early processing cost. It then progressively selects representatives using prompt attention under equal per-image quotas and aggregates unselected tokens into similar representatives from the same image. These operations keep every image represented and its tokens separate from other images, while refining the retained representations. A closed-form schedule sets retention counts to target a prescribed average token budget across decoder layers. Across four VLM backbones, SIEVE achieves the highest average at a 6.25% budget on five multi-image benchmarks and four single-image benchmarks, and the highest mean on two video benchmarks. On LLaVA-OneVision-7B, it outperforms the strongest baseline by 9.96 and 9.03 points at 6.25% and 12.5% budgets, and at 12.5% it exceeds the uncompressed model by 2.01 points with 6.6x fewer prefill FLOPs.
Figures & tables
| Method | Pre-decoder compression | Text-guided reduction | Progressive selection | Per-image coverage | Same-image aggregation |
| VisionZip [ 23 ] | ✓ | ✗ | ✗ | ✓ † | ✓ † |
| FastV [ 7 ] | ✗ | ✓ | ✗ | ✗ | ✗ |
| PyramidDrop [ 22 ] | ✗ | ✓ | ✓ | ✗ | ✗ |
| SparseVLM [ 28 ] | ✗ | ✓ | ✓ | ✗ | ✗ |
| AVTP [ 27 ] | ✗ | ✗ | ✓ | ✗ | ✗ |
| CLSE [ 5 ] | ✗ | ✗ | ✗ | ✗ | ✗ |
| Method | LLaVA-OneVision-7B | Qwen2.5-VL-7B | Qwen3-VL-8B | InternVL3-8B | |||||||||||||||||||||
| Venue | Mant. | NLVR2 | Muir | BLINK | QB2 | Avg | Mant. | NLVR2 | Muir | BLINK | QB2 | Avg | Mant. | NLVR2 | Muir | BLINK | QB2 | Avg | Mant. | NLVR2 | Muir | BLINK | QB2 | Avg | |
| Full | 60.83 | 89.22 | 41.96 | 49.87 | 73.70 | 63.12 | 75.12 | 87.90 | 58.58 | 57.44 | 75.20 | 70.85 | 66.82 | 87.18 | 50.31 | 58.86 | 76.40 | 67.91 | 68.20 | 85.72 | 54.04 | 56.34 | 77.10 | 68.28 | |
| Keep 6.25% of visual tokens | |||||||||||||||||||||||||
| VisionZip [ 23 ] | CVPR’25 | 51.15 | 66.47 | 40.08 | 41.40 | 56.10 | 51.04 | 65.44 | 70.22 | 43.58 | 45.71 | 66.30 | 58.25 | 57.60 | 66.86 | 43.19 | 38.03 | 68.20 | 54.78 | 59.45 | 72.74 | 45.38 | 46.29 | 68.10 | 58.39 |
| FastV [ 7 ] | ECCV’24 | 36.87 | 49.45 | 26.04 | 39.24 | 45.80 | 39.48 | 42.86 | 49.86 | 26.65 | 39.72 | 46.20 | 41.06 | 30.88 | 50.62 | 43.85 | 34.77 | 47.10 | 41.44 | 41.47 | 50.84 | 20.92 | 36.93 | 45.20 | 39.07 |
| PyramidDrop [ 22 ] | CVPR’25 | 36.87 | 49.45 | 26.04 | 39.24 | 45.80 | 39.48 | 42.86 | 49.86 | 26.65 | 39.72 | 46.20 | 41.06 | 30.88 | 50.62 | 43.85 | 34.77 | 47.10 | 41.44 | 41.47 | 50.84 | 20.92 | 36.93 | 45.20 | 39.07 |
| Method | FLOPs (T) | Storage (MB) | Prefill (ms) | Acc |
| Full | 26.46 | 107.4 | 100.4 | 63.12 |
| Keep 12.5% of visual tokens | ||||
| VisionZip [ 23 ] | 4.06 | 16.9 | 47.0 | 56.10 |
| FastV [ 7 ] | 4.13 | 16.9 | 47.9 | 41.16 |
| PyramidDrop [ 22 ] | 4.13 | 16.9 | 52.0 | 41.68 |
| SparseVLM [ 28 ] | 4.13 | 16.9 | 53.8 | 43.11 |
| Position | Joint evidence | |||||||
| Method | Mean | Worst | Mean | Worst | Joint | Joint | CF-A | CF-B |
| Full | 52.9 | 51.2 | 48.1 | 45.2 | 77.1 | 63.9 | 23.6 | 23.2 |
| VisionZip [ 23 ] | 43.1 | 41.2 | 40.1 | 37.8 | 67.2 | 60.7 | 20.5 | 19.8 |
| FastV [ 7 ] | 30.5 | 29.0 | 30.0 | 27.8 | 50.8 | 44.0 | 2.2 | 2.3 |
| PyramidDrop [ 22 ] | 28.8 | 27.0 | 28.6 | 26.8 | 49.9 | 42.0 | 2.7 | 3.2 |
| Method | LLaVA-OneVision-7B | Qwen2.5-VL-7B | Qwen3-VL-8B | InternVL3-8B | Avg |
| Full | 58.72 | 62.59 | 66.37 | 64.81 | 63.12 |
| Keep 6.25% of visual tokens | |||||
| VisionZip [ 23 ] | 56.74 | 54.86 | 57.49 | 55.44 | 56.13 |
| FastV [ 7 ] | 37.64 | 35.36 | 39.67 | 38.76 | 37.86 |
| PyramidDrop [ 22 ] | 37.64 | 35.36 | 39.75 | 38.76 | 37.88 |
| AVTP [ 27 ] | 39.00 | 48.52 | 46.86 | 46.08 | 45.12 |
| Method | LLaVA-OneVision-7B | Qwen2.5-VL-7B | Qwen3-VL-8B | InternVL3-8B | Avg |
| Full | 80.67 | 80.59 | 81.76 | 84.44 | 81.87 |
| Keep 6.25% of visual tokens | |||||
| VisionZip [ 23 ] | 65.33 | 55.48 | 54.17 | 61.05 | 59.01 |
| FastV [ 7 ] | 40.13 | 38.85 | 42.28 | 40.12 | 40.35 |
| PyramidDrop [ 22 ] | 40.13 | 38.85 | 42.28 | 40.12 | 40.35 |
| CLSE [ 5 ] | — | — | 42.67 | — | — |
| Mantis | NLVR2 | Muir | BLINK | Q-Bench2 | Avg | ||
| SIEVE | 63.59 | 83.61 | 47.77 | 45.50 | 70.50 | 62.19 | |
| (1) Cross-image coverage | |||||||
| Budget allocation (SIEVE: equal quota per image) | |||||||
| one global top- , no per-image quota | 59.91 | 82.17 | 47.96 | 40.93 | 69.60 | 60.11 | -2.08 |
| minimum per image, remainder by attention | 59.45 | 82.99 | 46.85 | 44.24 | 68.90 | 60.48 | -1.71 |
| Selection score (SIEVE: prompt attention) | |||||||
| Keep 6.25% | Keep 12.5% | |||
| OneVision | Qwen2.5 | OneVision | Qwen2.5 | |
| SIEVE | 62.19 | 63.67 | 65.13 | 66.21 |
| Recursive, | 58.05 | 60.75 | 63.74 | 63.76 |
| Recursive, | 57.89 | 60.59 | 64.03 | 63.09 |
| Recursive, | 55.81 | 58.08 | 62.01 | 61.90 |
| No pooling, | 46.51 | 56.14 | 51.75 | 63.77 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| relative depth | |||||||
| Backbone | 0.07 | 0.30 | 0.52 | 0.70 | 0.85 | 1.00 | |
| LLaVA-OneVision-7B | Overlap | 82.5 | 81.0 | 44.8 | 25.0 | 23.0 | 25.2 |
| Asked | 24.9 | 25.4 | 44.3 | 69.9 | 70.0 | 62.7 | |
| Qwen3-VL-8B | Overlap | 85.3 | 85.7 | 81.0 | 22.7 | 53.1 | 73.2 |
| Asked | 25.1 | 25.0 | 28.6 | 71.9 | 38.8 | 26.1 | |
| InternVL3-8B | Overlap | 84.4 | 82.6 | 67.0 | 31.7 | 35.7 | 35.7 |
| Method | TextVQA | SQA-I | MMStar | POPE | Avg | |||||||||||||||
| OV | Qwen2.5 | Qwen3 | IVL3 | OV | Qwen2.5 | Qwen3 | IVL3 | OV | Qwen2.5 | Qwen3 | IVL3 | OV | Qwen2.5 | Qwen3 | IVL3 | OV | Qwen2.5 | Qwen3 | IVL3 | |
| Full | 76.21 | 84.50 | 83.53 | 81.53 | 95.93 | 88.30 | 94.55 | 98.36 | 61.33 | 61.73 | 60.13 | 67.20 | 89.22 | 87.82 | 88.82 | 90.66 | 80.67 | 80.59 | 81.76 | 84.44 |
| Keep 6.25% of visual tokens | ||||||||||||||||||||
| VisionZip [ 23 ] | 49.93 | 21.92 | 21.65 | 24.03 | 84.48 | 79.92 | 78.58 | 86.12 | 44.27 | 40.00 | 37.47 | 47.20 | 82.63 | 80.10 | 78.98 | 86.86 | 65.33 | 55.48 | 54.17 | 61.05 |
| FastV [ 7 ] | 9.15 | 9.03 | 9.10 | 9.93 | 74.37 | 73.03 | 73.57 | 73.97 | 29.53 | 27.93 | 28.47 | 30.40 | 47.46 | 45.42 | 57.96 | 46.17 | 40.13 | 38.85 | 42.28 | 40.12 |
| PyramidDrop [ 22 ] | 9.15 | 9.03 | 9.10 | 9.93 | 74.37 | 73.03 | 73.57 | 73.97 | 29.53 | 27.93 | 28.47 | 30.40 | 47.46 | 45.42 | 57.96 | 46.17 | 40.13 | 38.85 | 42.28 | 40.12 |
| Method | LLaVA-OneVision | Qwen2.5-VL | Qwen3-VL | InternVL3 |
| Full | 47.90 | 63.14 | 52.65 | 56.11 |
| Keep 6.25% of visual tokens | ||||
| VisionZip [ 23 ] | 40.33 | 47.45 | 47.54 | 47.35 |
| FastV [ 7 ] | 26.00 | 29.47 | 46.44 | 18.98 |
| PyramidDrop [ 22 ] | 26.00 | 29.47 | 46.44 | 18.98 |
| AVTP [ 27 ] | 28.38 | 43.70 | 47.54 | 33.12 |
| LLaVA-OneVision-7B | Qwen2.5-VL-7B | Qwen3-VL-8B | InternVL3-8B | ||||||
| Method | V | E | V | E | V | E | V | E | Avg |
| Full | 53.63 | 63.80 | 56.78 | 68.40 | 60.93 | 71.80 | 59.81 | 69.80 | 63.12 |
| Keep 6.25% of visual tokens | |||||||||
| VisionZip [ 23 ] | 50.67 | 62.80 | 49.93 | 59.80 | 53.78 | 61.20 | 52.48 | 58.40 | 56.13 |
| FastV [ 7 ] | 39.07 | 36.20 | 40.33 | 30.40 | 41.74 | 37.60 | 41.52 | 36.00 | 37.86 |
| PyramidDrop [ 22 ] | 39.07 | 36.20 | 40.33 | 30.40 | 41.89 | 37.60 | 41.52 | 36.00 | 37.88 |
| Mantis | NLVR2 | Muir | BLINK | Q-Bench2 | Avg | ||
| SIEVE | 71.43 | 78.05 | 50.96 | 49.50 | 68.40 | 63.67 | |
| (1) Cross-image coverage | |||||||
| Budget allocation (SIEVE: equal quota per image) | |||||||
| one global top- , no per-image quota | 67.28 | 77.84 | 45.65 | 48.92 | 65.60 | 61.06 | -2.61 |
| size-proportional share per image | 70.05 | 77.31 | 45.77 | 49.34 | 65.40 | 61.57 | -2.10 |
| Selection score (SIEVE: prompt attention) | |||||||
| Mantis | NLVR2 | Muir | BLINK | Q-Bench2 | Avg | ||
| SIEVE | 60.83 | 73.20 | 47.27 | 44.29 | 68.50 | 58.82 | |
| (1) Cross-image coverage | |||||||
| Budget allocation (SIEVE: equal quota per image) | |||||||
| one global top- , no per-image quota | 59.45 | 72.05 | 43.81 | 44.29 | 67.30 | 57.38 | -1.44 |
| size-proportional share per image | 59.45 | 72.84 | 43.96 | 44.71 | 67.20 | 57.63 | -1.19 |
| Selection score (SIEVE: prompt attention) | |||||||
| Mantis | NLVR2 | Muir | BLINK | Q-Bench2 | Avg | ||
| SIEVE | 67.74 | 78.30 | 50.92 | 49.50 | 71.40 | 63.57 | |
| (1) Cross-image coverage | |||||||
| Budget allocation (SIEVE: equal quota per image) | |||||||
| one global top- , no per-image quota | 64.52 | 75.69 | 47.19 | 47.34 | 65.40 | 60.03 | -3.54 |
| size-proportional share per image | 65.90 | 75.21 | 46.65 | 47.29 | 66.10 | 60.23 | -3.34 |
| Selection score (SIEVE: prompt attention) | |||||||
| Method | Mean | Worst | Gap | Mean | Worst | Gap |
| Full | 52.9 | 51.2 | 2.6 | 48.1 | 45.2 | 7.2 |
| VisionZip [ 23 ] | 43.1 | 41.2 | 4.6 | 40.1 | 37.8 | 7.2 |
| FastV [ 7 ] | 30.5 | 29.0 | 5.0 | 30.0 | 27.8 | 9.2 |
| PyramidDrop [ 22 ] | 28.8 | 27.0 | 3.2 | 28.6 | 26.8 | 3.2 |
| SparseVLM [ 28 ] | 30.4 | 28.8 | 3.2 | 29.8 | 27.4 | 7.0 |
| Multi-image | Single-image | ||||||||||
| Mantis | NLVR2 | Muir | BLINK | Q-Bench2 | Avg | TextVQA | SQA-I | MMStar | POPE | Avg | |
| LLaVA-OneVision-7B , keep 6.25% of visual tokens | |||||||||||
| SIEVE (progressive) | 63.59 | 83.61 | 47.77 | 45.50 | 70.50 | 62.19 | 57.90 | 84.53 | 49.73 | 86.46 | 69.66 |
| recursive, | 61.29 | 79.09 | 46.15 | 40.50 | 63.20 | 58.05 | 52.32 | 83.49 | 48.33 | 84.06 | 67.05 |
| recursive, | 58.06 | 78.77 | 46.42 | 39.72 | 66.50 | 57.89 | 56.55 | 82.55 | 46.67 | 84.83 | 67.65 |
| recursive, | 54.84 | 75.53 | 43.92 | 40.35 | 64.40 | 55.81 | 53.52 | 81.71 | 45.27 | 84.09 | 66.15 |
| Method | Joint | Both | All 3 | Worst | CF-A | CF-B | Joint | Both | All 3 | Worst | CF-A | CF-B |
| Full | 77.1 | 77.6 | 62.4 | 70.4 | 47.2 | 42.0 | 63.9 | 62.1 | 40.5 | 49.0 | 23.6 | 23.2 |
| VisionZip [ 23 ] | 67.2 | 57.7 | 40.9 | 57.6 | 29.4 | 26.1 | 60.7 | 56.7 | 34.9 | 47.6 | 20.5 | 19.8 |
| FastV [ 7 ] | 50.8 | 24.8 | 20.9 | 41.2 | 2.3 | 1.7 | 44.0 | 19.0 | 12.8 | 36.8 | 2.2 | 2.3 |
| PyramidDrop [ 22 ] | 49.9 | 24.0 | 16.7 | 44.4 | 1.3 | 1.0 | 42.0 | 17.1 | 8.8 | 36.0 | 2.7 | 3.2 |
| SparseVLM [ 28 ] | 51.9 | 29.8 | 24.4 | 45.2 | 2.1 | 2.2 | 43.7 | 21.9 | 13.6 | 37.4 | 4.2 | 4.9 |
Efficient VLM Inference
Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.
See all 512 papers