C-Routing: Complementary Cropping and Compression for Efficient Vision-Language Inference
Organizations: Leiden University
Abstract
Visual Question Answering (VQA) imposes substantial computational overhead on vision-language models (VLMs), motivating model-agnostic input reduction methods that can be readily applied across diverse VLMs, including in black-box settings. Simply enforcing a visual-token cap reduces computation, but it can discard information needed to answer the question. A single reduction strategy also struggles to accommodate different scenarios. Our key observation is that cropping and compression are complementary: cropping highlights small targets and local details, whereas whole-image compression better preserves context for large targets or distributed evidence. Based on this observation, we propose C-Routing, which adaptively selects cropping or compression according to the input image and question. Since both strategies may still discard key information, we further design a safety gate that uses the predicted joint correctness distribution under original and reduced inputs to guide fallback and protect accuracy. To accommodate consumer GPU resource constraints, all training and evaluation experiments run on RTX 3090 GPUs. To measure gains beyond token capping, we use full-image baselines with the same cap of 1,024 visual tokens. We conduct extensive evaluation on three VLMs and 18 datasets spanning local details, global scenes, scene text, and documents. With the visual budget already capped, C-Routing further reduces end-to-end FLOPs by 36.53% and visual tokens by 42.36% on average, with a mean relative task-score decrease of only 0.77%.
Figures & tables
| Qwen2.5-VL-7B | Qwen2.5-VL-3B | InternVL3-8B | |||||||||||||
| Method | Score | Retention (%) | (pp) | FLOPs saved (%) | Tokens saved (%) | Score | Retention (%) | (pp) | FLOPs saved (%) | Tokens saved (%) | Score | Retention (%) | (pp) | FLOPs saved (%) | Tokens saved (%) |
| 67.03 | 100.00 | +0.000 | 0.00 | 0.00 | 66.60 | 100.00 | +0.000 | 0.00 | 0.00 | 66.89 | 100.00 | +0.000 | 0.00 | 0.00 | |
| Accuracy-oriented references | |||||||||||||||
| ViCrop 256 [ 38 ] | 63.47 | 94.36 | -3.556 | -20.65 | 56.33 | 62.56 | 94.69 | -4.038 | -17.47 | 56.31 | N/A | ||||
| ViCrop 1024 | 67.99 | 102.21 | +0.967 | -203.10 | -5.40 | 67.08 | 101.36 | +0.487 | -203.26 | -5.20 | N/A | ||||
| ZoomEye [ 29 ] | 69.09 | 105.55 | +2.066 | -2328.90 | 9.68 | 68.80 | 106.91 | +2.202 | -4603.66 | 16.35 | 70.12 | 108.63 | +3.227 | -5278.93 | 0.95 |
| Score change (pp) / FLOPs saving (%) | |||
| Target | Full router | No crop | No compression |
| 7B | -0.535 / 39.09 | -1.994 / 39.81 | +0.867 / 9.01 |
| 3B | -0.862 / 35.18 | -1.185 / 34.38 | +0.567 / -0.09 |
| InternVL3-8B | +0.104 / 37.75 | -0.014 / 37.93 | +0.680 / -3.60 |
| Gate | 7B | 3B | InternVL3-8B | Mean (3 models) |
| Balanced-A | ||||
| None | -1.206 / 42.33 | -1.986 / 43.51 | -1.556 / 42.76 | -1.583 / 42.87 |
| Crop | -1.292 / 40.93 | -1.933 / 42.50 | -1.571 / 42.72 | -1.599 / 42.05 |
| Compression | -0.587 / 37.87 | -0.868 / 36.33 | +0.116 / 37.84 | -0.446 / 37.35 |
| Both | -0.673 / 36.47 | -0.815 / 35.31 | +0.101 / 37.81 | -0.462 / 36.53 |
| Safe-A | ||||
| Stage shares (%) | FLOPs savings (%) | |||||
| Target | Shared features | Grounding DINO | Prediction heads | Final answer | Final answer | End-to-end |
| 7B | 2.60 | 0.68 | 96.71 | 39.60 | 36.47 | |
| 3B | 3.95 | 0.71 | 95.34 | 40.20 | 35.31 | |
| InternVL3-8B | 3.04 | 0.01 | 96.94 | 40.02 | 37.81 | |
| Method | score (pp) | FLOPs saved (%) | Tokens saved (%) |
| Original input | +0.00 | 0.00 | 0.00 |
| Ours | -0.55 | 39.09 | 44.27 |
| VisionZip | -0.28 | 20.39 | 30.13 |
| Ours + VisionZip (direct) | -1.38 | 50.29 | 61.13 |
| Ours + VisionZip (adapted) | +0.37 | 29.44 | 42.15 |
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Target | Setting | Validation met | Score D / M | Retention (%) D / M | pp D / M | FLOPs saving (%) D / M | Token saving (%) D / M |
| 7B | Safe-M | 4/4 | 66.79 / 67.39 | 99.44 / 99.48 | -0.234 / -0.225 | 25.35 / 24.92 | 30.42 / 30.02 |
| 7B | Safe-A | 4/4 | 66.84 / 67.43 | 99.48 / 99.52 | -0.189 / -0.184 | 25.32 / 24.87 | 30.52 / 30.10 |
| 7B | Balanced-M | 4/4 | 66.11 / 66.71 | 98.40 / 98.47 | -0.911 / -0.903 | 37.39 / 37.04 | 42.34 / 41.99 |
| 7B | Balanced-A | 4/4 | 66.35 / 66.92 | 98.78 / 98.81 | -0.673 / -0.692 | 36.47 / 36.25 | 41.67 / 41.46 |
| 3B | Safe-M | 4/4 | 66.70 / 67.35 | 100.19 / 100.11 | +0.099 / +0.044 | 21.52 / 20.80 | 28.92 / 28.29 |
| 3B | Safe-A | 4/4 | 66.59 / 67.28 | 99.93 / 99.89 | -0.002 / -0.034 | 19.35 / 18.69 | 26.03 / 25.46 |
| Qwen2.5-VL-7B | Qwen2.5-VL-3B | InternVL3-8B | Target mean | |||||
| Method | Score | Saving | Score | Saving | Score | Saving | pp | Saving |
| 67.03 | 0.00 | 66.60 | 0.00 | 66.89 | 0.00 | +0.000 | 0.00 | |
| 65.59 | 17.46 | 64.63 | 14.29 | 65.37 | 2.01 | -1.639 | 11.25 | |
| 62.26 | 63.10 | 60.77 | 64.50 | 63.79 | 55.56 | -4.564 | 61.05 | |
| 65.00 | 38.81 | 64.60 | 39.80 | 65.24 | 36.64 | -1.891 | 38.42 | |
| Qwen2.5-VL-7B | Qwen2.5-VL-3B | InternVL3-8B | |||||||
| Method | Retention | FLOPs saved | Tokens saved | Retention | FLOPs saved | Tokens saved | Retention | FLOPs saved | Tokens saved |
| All 18 datasets | |||||||||
| CARES-F1024 [ 13 ] | 97.76 | 35.75 | 39.39 | 96.51 | 32.93 | 37.01 | 98.75 | 29.82 | 34.85 |
| CropVLM-F1024 [ 3 ] | 97.41 | -36.17 | -8.78 | 97.68 | -56.58 | -8.78 | 98.39 | -49.65 | -21.62 |
| ViCrop 256 [ 38 ] | 94.21 | -21.22 | 56.06 | 94.25 | -18.08 | 56.04 | N/A | ||
| ViCrop 1024 | 101.97 | -203.13 | -5.43 | 101.23 | -203.30 | -5.24 | N/A | ||
| Target | Method | Score D / M | pp D / M | FLOPs saving (%) D / M | Token saving (%) D / M |
| 7B | 67.03 / 67.61 | +0.000 / +0.000 | 0.00 / 0.00 | 0.00 / 0.00 | |
| 7B | CARES-F1024 [ 13 ] | 65.65 / 66.22 | -1.373 / -1.391 | 35.69 / 35.75 | 39.36 / 39.39 |
| 3B | 66.60 / 67.31 | +0.000 / +0.000 | 0.00 / 0.00 | 0.00 / 0.00 | |
| 3B | CARES-F1024 [ 13 ] | 64.33 / 65.03 | -2.265 / -2.277 | 33.03 / 32.93 | 37.12 / 37.01 |
| InternVL3-8B | 66.89 / 67.34 | +0.000 / +0.000 | 0.00 / 0.00 | 0.00 / 0.00 | |
| InternVL3-8B | CARES-F1024 [ 13 ] | 65.81 / 66.15 | -1.077 / -1.189 | 29.89 / 29.82 | 34.98 / 34.85 |
| (a) Datasets 1–9 | ||||||||||||||||||
| AI2D | ChartQA | DocVQA | HR-Bench 8K | InfoVQA | MME | MMStar | OCRBench | Pinpoint- Bench | ||||||||||
| Method | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved |
| Qwen2.5-VL-7B | ||||||||||||||||||
| 81.25 | 0.00 | 76.51 | 0.00 | 92.83 | 0.00 | 53.72 | 0.00 | 71.17 | 0.00 | 84.54 | 0.00 | 60.45 | 0.00 | 73.51 | 0.00 | 27.08 | 0.00 | |
| Safe-M | 81.89 | 34.28 | 75.83 | 19.52 | 91.69 | 22.56 | 53.46 | 23.52 | 71.32 | -0.26 | 83.97 | 41.53 | 61.37 | 14.98 | 74.32 | 18.64 | 25.39 | 35.73 |
| Safe-A | 81.90 | 33.11 | 75.81 | 18.48 | 91.69 | 22.54 | 53.66 | 22.84 | 71.32 | -0.26 | 83.94 | 41.66 | 61.32 | 14.14 | 74.44 | 18.44 | 24.87 | 35.51 |
| (b) Datasets 10–18 | ||||||||||||||||||
| POPE | RealWorld QA | ScienceQA | SEED-Bench Image | SOU | TextVQA | VizWiz | V ∗ | ZoomBench | ||||||||||
| Method | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved |
| Qwen2.5-VL-7B | ||||||||||||||||||
| 85.54 | 0.00 | 62.75 | 0.00 | 82.89 | 0.00 | 76.53 | 0.00 | 53.31 | 0.00 | 80.53 | 0.00 | 42.50 | 0.00 | 66.67 | 0.00 | 34.68 | 0.00 | |
| Safe-M | 85.85 | 20.67 | 61.19 | 19.19 | 81.98 | 34.01 | 75.77 | 30.63 | 53.33 | 32.04 | 79.13 | 17.91 | 42.50 | 43.39 | 69.44 | 25.13 | 33.82 | 22.84 |
| Safe-A | 85.77 | 20.62 | 61.19 | 20.94 | 82.11 | 34.18 | 75.79 | 30.43 | 53.45 | 31.88 | 79.17 | 17.51 | 42.49 | 43.30 | 69.79 | 27.50 | 34.32 | 23.00 |
| CARES | ZoomEye final-cap | ZoomEye fallback | |||||||||||
| Dataset | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved | |
| AI2D | 1365 | 77.58 | 15.54 | 81.61 | 51.00 | 81.61 | 27.08 | 81.90 | 30.21 | 81.17 | -767.30 | 81.17 | -774.10 |
| ChartQA | 13038 | 69.81 | 11.33 | 71.84 | 55.92 | 75.93 | 21.28 | 75.73 | 29.98 | 76.50 | -361.47 | 76.47 | -362.36 |
| DocVQA | 17484 | 86.56 | 1.08 | 76.69 | 73.52 | 89.11 | 50.32 | 87.61 | 58.65 | 91.89 | -982.04 | 91.79 | -981.99 |
| HR-Bench 8K | 376 | 61.17 | 4.14 | 46.01 | 73.18 | 51.06 | 48.65 | 50.27 | 45.92 | 61.97 | -11747.76 | 61.97 | -11747.49 |
| InfoVQA | 1098 | 54.16 | 17.86 | 48.81 | 72.44 | 63.83 | 47.06 | 65.77 | 39.80 | 71.20 | -1218.11 | 71.20 | -1218.02 |
| CARES | ZoomEye final-cap | ZoomEye fallback | |||||||||||
| Dataset | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved | |
| AI2D | 1365 | 76.04 | 10.97 | 79.27 | 52.61 | 79.63 | 28.03 | 79.34 | 28.28 | 79.19 | -1974.90 | 79.05 | -1975.35 |
| ChartQA | 13038 | 76.02 | 7.70 | 76.56 | 57.47 | 83.08 | 21.97 | 81.63 | 37.46 | 82.31 | -2018.76 | 82.30 | -2018.83 |
| DocVQA | 17484 | 89.18 | -1.17 | 70.80 | 74.52 | 88.17 | 51.20 | 89.48 | 50.97 | 90.59 | -4603.96 | 90.58 | -4603.96 |
| HR-Bench 8K | 376 | 54.52 | 1.62 | 44.41 | 74.51 | 52.13 | 49.75 | 47.87 | 44.97 | 64.10 | -14299.69 | 64.10 | -14299.69 |
| InfoVQA | 1098 | 52.59 | 16.16 | 42.46 | 73.46 | 57.58 | 47.93 | 63.74 | 27.06 | 65.81 | -4666.85 | 65.81 | -4666.85 |
| CARES | ZoomEye final-cap | ZoomEye fallback | |||||||||||
| Dataset | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved | Score | Saved | |
| AI2D | 1365 | 78.17 | -1.72 | 82.20 | 45.03 | 81.83 | 28.13 | 81.83 | 25.36 | 82.27 | -1656.13 | 82.27 | -1656.13 |
| ChartQA | 13038 | 69.36 | 16.31 | 71.85 | 67.89 | 72.96 | 66.04 | 72.96 | 64.52 | 73.66 | -965.83 | 73.66 | -965.83 |
| DocVQA | 17484 | 85.98 | -0.94 | 68.55 | 68.61 | 71.48 | 64.03 | 78.01 | 51.29 | 85.41 | -4467.92 | 85.40 | -4467.92 |
| HR-Bench 8K | 376 | 57.45 | -12.76 | 44.15 | 54.74 | 50.53 | 27.92 | 49.73 | 24.65 | 74.20 | -21536.65 | 74.20 | -21536.65 |
| InfoVQA | 1098 | 48.82 | 9.22 | 37.01 | 63.56 | 49.94 | 40.32 | 56.40 | 20.11 | 59.41 | -5435.88 | 59.41 | -5435.87 |
| Qwen2.5-VL-7B | Qwen2.5-VL-3B | InternVL3-8B | |||||||
| Dataset | Score | FLOPs saved | Tokens saved | Score | FLOPs saved | Tokens saved | Score | FLOPs saved | Tokens saved |
| AI2D | 81.03 | -71.51 | -31.65 | 80.07 | -102.84 | -31.65 | 83.15 | -72.12 | -35.65 |
| ChartQA | 77.85 | -56.84 | -27.96 | 82.82 | -79.41 | -27.96 | 70.98 | 8.13 | 32.64 |
| DocVQA | 89.53 | -8.75 | 13.85 | 89.99 | -23.83 | 13.85 | 76.90 | -8.96 | 19.70 |
| HR-Bench 8K | 50.80 | -17.70 | 0.31 | 53.46 | -30.08 | 0.31 | 50.27 | -62.40 | -35.66 |
| InfoVQA | 63.81 | -13.55 | 1.31 | 57.97 | -23.63 | 1.31 | 52.65 | -33.18 | -14.22 |
| Method | Mean | Best | Best dataset | Worst | Worst dataset | SD |
| Qwen2.5-VL-7B | ||||||
| CropVLM-F1024 [ 3 ] | -1.62 | +2.27 | MME | -7.36 | InfoVQA | 2.79 |
| CARES-F1024 [ 13 ] | -1.37 | +2.09 | MMStar | -5.40 | InfoVQA | 2.36 |
| ViCrop 256 [ 38 ] | -3.56 | +2.16 | MME | -16.16 | InfoVQA | 4.85 |
| ViCrop 1024 | +0.97 | +7.80 | ZoomBench | -2.02 | TextVQA | 2.33 |
| ZoomEye [ 29 ] | +2.07 | +13.54 | Pinpoint-Bench | -1.04 | TextVQA | 4.17 |
| Dataset macro | Mechanism macro | ||||||||
| Target | Configuration | Score (%) | (pp) | FLOPs saved (%) | Tokens saved (%) | Score (%) | (pp) | FLOPs saved (%) | Tokens saved (%) |
| Qwen2.5-VL-7B | Balanced-A | ||||||||
| Balanced-M | |||||||||
| Safe-A | |||||||||
| Safe-M | |||||||||
| Qwen2.5-VL-3B | Balanced-A | ||||||||
| 7B | 3B | InternVL3-8B | ||||||||
| Setting | Branches | Validation | score (pp) | FLOPs (%) | Validation | score (pp) | FLOPs (%) | Validation | score (pp) | FLOPs (%) |
| Safe-M | Both | Yes | -0.313 | 28.96 | Yes | +0.097 | 21.21 | No | +1.019 | 25.11 |
| Safe-M | Compression | No | -0.922 | 28.86 | No | -0.487 | 23.92 | No | +0.611 | 25.81 |
| Safe-M | Crop | No | +0.927 | 3.90 | No | +0.638 | -1.43 | Yes | +0.611 | -4.58 |
| Safe-A | Both | Yes | -0.115 | 27.64 | Yes | -0.111 | 21.67 | Yes | +0.849 | 25.47 |
| Safe-A | Compression | No | -0.922 | 28.86 | Yes | -0.429 | 21.26 | No | +0.611 | 25.81 |
| Dataset macro | Mechanism macro | |||||||||
| Setting | Safety | Feasible | Score | pp | FLOPs (%) | Tokens (%) | Score | pp | FLOPs (%) | Tokens (%) |
| Qwen2.5-VL-7B | ||||||||||
| Safe-M | None | 0/4 | 65.93 0.27 | -1.092 | 42.29 2.80 | 48.00 | 66.42 0.29 | -1.194 | 42.33 2.78 | 48.07 |
| Safe-M | Crop | 0/4 | 65.75 0.45 | -1.274 | 38.79 2.48 | 44.30 | 66.27 0.45 | -1.344 | 38.83 2.41 | 44.36 |
| Safe-M | Compression | 0/4 | 66.97 0.14 | -0.052 | 28.85 4.89 | 34.13 | 67.54 0.12 | -0.075 | 28.42 4.98 | 33.73 |
| Safe-M | Both | 4/4 | 66.79 0.33 | -0.234 | 25.35 5.24 | 30.42 | 67.39 0.30 | -0.225 | 24.92 5.36 | 30.02 |
| Dataset | Questions | 16,384-cap score | 1,024-cap score | score (pp) | FLOPs saved (%) | Tokens saved (%) |
| V* | 72 | 75.00 | 70.83 | -4.17 | 79.71 | 77.63 |
| DocVQA | 200 | 98.94 | 99.10 | +0.16 | 80.62 | 78.06 |
| RealWorldQA | 200 | 68.00 | 69.50 | +1.50 | 46.48 | 45.80 |
| AI2D | 200 | 79.50 | 79.50 | +0.00 | 15.76 | 15.78 |
| Dataset macro | 672 | 80.36 | 79.73 | -0.63 | 55.64 | 54.32 |
| Mechanism macro | 672 | 82.56 | 81.48 | -1.09 | 63.81 | 62.16 |
| Dataset | Token cap | Visual tokens | TFLOPs/query | Allocated GiB | Reserved GiB |
| V* | 16,384 | 4499.25 | 98.01 | 17.07 | 18.71 |
| V* | 1,024 | 1006.49 | 19.89 | 15.74 | 16.13 |
| DocVQA | 16,384 | 4456.81 | 99.09 | 18.81 | 22.04 |
| DocVQA | 1,024 | 977.99 | 19.21 | 15.74 | 16.13 |
| RealWorldQA | 16,384 | 1724.49 | 34.65 | 16.01 | 16.62 |
| RealWorldQA | 1,024 | 934.64 | 18.55 | 15.75 | 16.13 |
| 7B | 3B | InternVL3-8B | ||||
| Stage | TFLOPs/query | Share (%) | TFLOPs/query | Share (%) | TFLOPs/query | Share (%) |
| Ours Balanced-A | ||||||
| Shared features | 0.258 | 2.60 | 0.258 | 3.95 | 0.258 | 3.04 |
| Selection head | ||||||
| Crop constructor | 0.068 | 0.68 | 0.047 | 0.71 | 0.001 | 0.01 |
| Resolution head | ||||||
| Target | Method | General attn. | Question attn. | Cue generation/filter | Search | Final answer |
| 7B | ViCrop 256 [ 38 ] | 30.01 | 31.90 | 0 | 0 | 38.09 |
| 7B | ViCrop 1024 | 32.54 | 33.18 | 0 | 0 | 34.28 |
| 3B | ViCrop 256 | 30.26 | 31.74 | 0 | 0 | 38.00 |
| 3B | ViCrop 1024 | 32.65 | 33.14 | 0 | 0 | 34.20 |
| 7B | ZoomEye official final-cap [ 29 ] | 0 | 0 | 4.42 | 87.25 | 8.34 |
| 7B | ZoomEye matched-F fallback | 0 | 0 | 4.41 | 87.19 | 8.39 |
| score | Final-answer inference | Saving vs. own original input (%) | End-to-end | ||||
| Method | (pp; D / M) | TFLOPs/query | Vis. tokens | Final-answer FLOPs (D / M) | Vis. tokens (D / M) | End-to-end FLOPs (D / M) | TFLOPs/query |
| Qwen2.5-VL-7B | |||||||
| Original | 0.00 / 0.00 | 15.17 | 757.1 | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | 15.17 |
| Crop | -1.43 / -1.71 | 11.96 | 586.0 | 22.70 / 22.72 | 24.50 / 24.51 | 17.46 / 17.48 | 12.65 |
| Compress 256 | -4.77 / -4.93 | 4.99 | 224.0 | 63.10 / 62.93 | 66.49 / 66.27 | 63.10 / 62.93 | 4.99 |
| Compress 512 | -2.02 / -2.01 | 8.69 | 421.2 | 38.81 / 38.71 | 40.47 / 40.34 | 38.81 / 38.71 | 8.69 |
| Target | Visual-token cap | Out of memory / attempted | Allocated GiB | Reserved GiB |
| Qwen2.5-VL-3B | 16,384 | 22 / 36 | 16.01 | 18.37 |
| Qwen2.5-VL-3B | 1,024 | 0 / 36 | 9.70 | 10.64 |
| Qwen2.5-VL-7B | 16,384 | 25 / 36 | 18.07 | 18.96 |
| Qwen2.5-VL-7B | 1,024 | 0 / 36 | 18.09 | 18.96 |
| InternVL3-8B | 3,072 | 0 / 36 | 17.88 | 19.23 |
| InternVL3-8B | 1,024 | 0 / 36 | 15.34 | 15.61 |
| (a) Aggregate latency | |||||
| Method | Frontend (ms) | Target (ms) | E2E (ms) | Savings (%) | 95% CI (%) |
| 0.01 | 765.32 | 765.33 | – | – | |
| CARES- | 115.68 | 502.65 | 618.33 | 18.25 | [14.61, 21.36] |
| Ours Balanced-A | 167.45 | 491.53 | 658.98 | 13.72 | [9.47, 17.52] |
| (b) Per-dataset latency | |||||
| Dataset | F1024 E2E | CARES E2E | Ours E2E | Ours frontend | Savings (%) CARES / Ours |
| Ours | VisionZip | Ours + VisionZip (direct) | ||||||||
| Dataset | Original score | FLOPs | Tokens | FLOPs | Tokens | FLOPs | Tokens | |||
| 70% visual tokens retained | ||||||||||
| AI2D | 81.25 | +0.37 | 35.38 | 41.40 | +0.44 | 19.87 | 30.18 | +0.95 | 46.91 | 59.16 |
| ChartQA | 76.32 | -0.99 | 20.22 | 24.52 | -0.60 | 19.96 | 30.05 | -2.39 | 35.26 | 47.33 |
| DocVQA | 92.39 | -3.57 | 49.10 | 51.18 | -0.92 | 21.08 | 30.09 | -8.12 | 59.18 | 65.93 |
| HR-Bench 8K | 53.72 | -4.26 | 46.96 | 50.05 | +1.86 | 20.80 | 30.12 | -4.79 | 57.14 | 65.14 |
| Ours | VisionZip | Ours + VisionZip (direct) | ||||||||
| Dataset | Original score | FLOPs | Tokens | FLOPs | Tokens | FLOPs | Tokens | |||
| 50% visual tokens retained | ||||||||||
| AI2D | 81.25 | +0.37 | 35.38 | 41.40 | -0.07 | 32.91 | 50.18 | +0.15 | 54.47 | 70.87 |
| ChartQA | 76.32 | -0.99 | 20.22 | 24.52 | -2.38 | 33.13 | 50.04 | -5.02 | 45.12 | 62.41 |
| DocVQA | 92.39 | -3.57 | 49.10 | 51.18 | -5.01 | 34.91 | 50.10 | -18.91 | 65.82 | 75.69 |
| HR-Bench 8K | 53.72 | -4.26 | 46.96 | 50.05 | +1.60 | 34.40 | 50.12 | -5.05 | 63.84 | 75.12 |
| Method | Retained tokens (%) | score (pp) | FLOPs saved (%) | Tokens saved (%) |
| Same-data router | 100 | -1.11 | 39.41 | 43.97 |
| Adapted combination | 70 | +0.37 | 29.44 | 42.15 |
| Adapted combination | 50 | -1.54 | 36.58 | 54.92 |
| Same-data router 100% | Adapted 70% | Adapted 50% | ||||||||
| Dataset | Original score | FLOPs | Tokens | FLOPs | Tokens | FLOPs | Tokens | |||
| AI2D | 81.25 | +0.29 | 47.64 | 54.64 | +0.95 | 37.07 | 49.81 | -0.29 | 31.43 | 51.37 |
| ChartQA | 76.32 | -3.11 | 43.98 | 49.67 | -0.66 | 15.91 | 30.17 | -2.58 | 32.21 | 51.15 |
| DocVQA | 92.39 | -3.84 | 29.73 | 31.76 | -0.92 | 19.79 | 30.09 | -5.02 | 33.65 | 50.12 |
| HR-Bench 8K | 53.72 | -4.26 | 44.45 | 47.46 | +6.65 | 29.27 | 39.53 | +1.60 | 33.00 | 50.12 |
| InfoVQA | 70.98 | -0.37 | 1.81 | 3.29 | -3.51 | 19.80 | 30.26 | -11.43 | 33.49 | 50.11 |
| Dataset mean | Task-group mean | ||||||
| Config | Gate | score (pp) | FLOPs saved (%) | Tokens saved (%) | score (pp) | FLOPs saved (%) | Tokens saved (%) |
| 100% visual tokens retained | |||||||
| Safe-M | None | -0.822 | 32.99 | 40.36 | -1.012 | 33.35 | 40.71 |
| Safe-M | Crop | -0.238 | 26.44 | 33.31 | -0.367 | 26.79 | 33.67 |
| Safe-M | Compression | -0.062 | 25.69 | 32.85 | -0.178 | 25.37 | 32.53 |
| Safe-M | Both | +0.522 | 19.13 | 25.81 | +0.468 | 18.81 | 25.49 |
| Dataset mean | Task-group mean | ||||||
| Config | Gate | score (pp) | FLOPs saved (%) | Tokens saved (%) | score (pp) | FLOPs saved (%) | Tokens saved (%) |
| 70% visual tokens retained | |||||||
| Safe-M | None | -1.754 | 50.69 | 61.64 | -2.000 | 50.71 | 61.70 |
| Safe-M | Crop | -1.899 | 47.64 | 58.73 | -2.094 | 47.59 | 58.72 |
| Safe-M | Compression | +0.422 | 32.73 | 44.97 | +0.291 | 32.34 | 44.67 |
| Safe-M | Both | +0.277 | 29.67 | 42.06 | +0.198 | 29.22 | 41.70 |
| Dataset | Training | Validation | Test | Metric |
| AI2D [ 12 ] | 1,105 | 618 | 1,365 | Multiple-choice accuracy |
| ChartQA [ 23 ] | 12,984 | 6,677 | 13,038 | Relaxed accuracy |
| DocVQA [ 24 ] | 19,291 | 8,037 | 17,484 | ANLS |
| HR-Bench 8K [ 31 ] | 300 | 124 | 376 | Multiple-choice accuracy |
| InfoVQA [ 25 ] | 1,110 | 593 | 1,098 | ANLS |
| MME [ 7 ] | 938 | 466 | 970 | Yes/no accuracy |
Efficient VLM Inference
Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.
See all 509 papers