cs.CVOct 13, 2026
SaveJust Respect the Semantics: Image-Token Compression via Pre-LLM Semantic Variable-Resolution Tokenization and In-LLM Relevance Pruning
Organizations: University of Toronto, Toronto, ON, Canada · Vector Institute, Toronto, ON, Canada · McGill University, Montreal, QC, Canada · University Health Network, Toronto, ON, Canada ·
Abstract
Image-token compression must reduce computation without hurting the semantics. We introduce Varest, guided by semantic fungibility: tokens within a coherent semantic region can be summarized coarsely, while object boundaries and literal text require finer distinctions. Object and text localizers guide variable-resolution merging, preserving spatial coverage and finer detail at boundaries and text. Learned [REL] registers then prune visual tokens after early decoder processing. Training across budgets enables one checkpoint per backbone. Across three backbones and 13 benchmarks, Varest outperforms 15 baselines in mean performance retention at every evaluated budget, retaining 86.10-92.33% of full-token performance with only 5.56% of the original layer-averaged image-token count.
Figures & tables
Figure 1: Per-benchmark performance of Varest across backbones and token budgets. General VQA benchmarks stay near or above full-token under compression, while text-dense benchmarks benefit most from larger budgets.
Figure 2: Overview of Varest. Object boundaries and textness are passed through the Varest tokenizer with post-projector visual tokens, which are then partitioned into semantic regions and each region is merged into one image token. At layer , 4 learned [REL] registers (the figure is not to scale) produce a score for each image token and keep the image tokens that are most relevant to LLM reading. Optional adaptive halting removes all remaining image tokens once the information horizon has been detected.
Figure 3: Visualized comparison of the baselines and Varest on LLaVA-1.5-7B. HiDrop and LearnPruner retain text but lose contextual information, while CORE’s coarse regions lose text. Varest keeps text cells at native resolution and separates the two people with object boundaries.
Figure 4: Compression signals and stages. Arrows progress toward explicit semantic priors. Varest preserves semantic structure and text detail before pruning inside the LLM.
Figure 5: Visualization of tokenization and [REL] pruning across budgets on LLaVA-1.5-7B. For each , the left panel shows the tokenization (text tokens in orange) and the right panel shows the tokens kept by [REL] . Text remains identifiable at every budget, while semantic granularity becomes finer as increases.
Figure 6: Benchmark-averaged performance retention versus layer-averaged image-token budget. 100% denotes uncompressed-backbone performance. Varest achieves the highest retention at every evaluated budget across all three backbones, preserving over 85% performance at on all three VLMs.
Figure 7: [REL] pruning behaviour under different prompts on LLaVA-1.5-7B. Varest keeps nearly the same tokens for two unrelated questions, yet answers both correctly, even when the named object is discarded.
| ToMe | VisionZip | VisionZip++ | DivPrune | VFlowOpt | OTPrune | ApET | FastV | SparseVLM | PyramidDrop | Nüwa | DUET-VLM | LearnPruner | HiDrop | CORE | Varest | |
| [ 4 ] | [ 35 ] | [ 35 ] | [ 1 ] | [ 36 ] | [ 7 ] | [ 20 ] | [ 5 ] | [ 38 ] | [ 34 ] | [ 10 ] | [ 28 ] | [ 30 ] | [ 32 ] | [ 15 ] | ||
| [ICLR2023] | [CVPR2025] | [CVPR2025] | [CVPR2025] | [ICCV2025] | [CVPR2026] | [CVPR2026] | [ECCV2024] | [ICML2025] | [CVPR2025] | [ICLR2026] | [CVPR2026] | [ICLR2026] | [ICLR2026] | [CVPR2026] | (Ours) | |
| LLaVA-1.5-7B | ||||||||||||||||
| 224 | 78.50% | 80.36% | 89.86% | 80.19% | 81.60% | 81.17% | 83.05% | 86.19% | 79.99% | 85.90% | 96.58% | 100.74% | 88.51% | 95.36% | 68.58% | 101.89% |
| 128 | 75.34% | 76.49% | 83.44% | 76.93% | 76.80% | 77.65% | 76.72% | 73.12% | 77.42% | 70.99% | 92.04% | 94.50% | 79.31% | 93.18% | 66.27% | 100.37% |
| 64 | 72.03% | 71.41% | 76.49% | 74.42% | 71.16% | 73.95% | 73.14% | 42.39% | 69.25% | 65.76% | 84.11% | 87.44% | 75.91% | 92.49% | 62.35% | 97.08% |
Table 1: Benchmark-averaged retention rates. We evaluate the average retention rate with respect to the uncompressed model of each method on 13 benchmarks across VQA, OCR, and grounding. Varest outperforms all 15 baselines at every budget across all three backbones, retaining – of baseline performance at the lowest budgets.
Figure 8: Performance retention versus measured computation. Each point averages both axes independently over four budgets; higher and further left is better. Varest achieves higher retention with slightly less computation than HiDrop, the strongest baseline by mean retention. Adaptive halting (AH) further reduces computation with a small retention loss.
| Backbone | Score | Trigger Rate | Skips [REL] | Tokens Saved |
|---|---|---|---|---|
| LLaVA-1.5-7B | 68.8% | 1.3% | 8.2% | |
| LLaVA-1.5-13B | 82.2% | 20.3% | 10.2% | |
| LLaVA-NeXT-8B | 84.9% | 3.9% | 6.9% |
Table 2: Optional halting on three backbones, averaged across benchmarks and budgets. The accuracy costs are relatively small compared to the additional savings.
| LLaVA-1.5-7B | LLaVA-1.5-13B | LLaVA-NeXT-8B | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TextVQA | DocVQA | ChartQA | CountBench | TextVQA | DocVQA | ChartQA | CountBench | TextVQA | DocVQA | ChartQA | CountBench | |||
| 224 | 224 | 1120 | ||||||||||||
| 128 | 128 | 640 | ||||||||||||
| 64 | 64 | 320 | ||||||||||||
| 32 | 32 | 160 | ||||||||||||
Table 3: Effect of removing the text localizer on selected benchmarks. Entries are changes in native 0–100 scores relative to full Varest at each token budget, with textness set to zero without retraining. The absence of the text localizer damages TextVQA most consistently on all three backbones and DocVQA on LLaVA-1.5. CountBench is extremely damaged on LLaVA-1.5-7B and slightly on LLaVA-NeXT-8B.
| Varest | - [REL] | Random Selection † | Oracle Selection † | -Object Localizer † | -Text Localizer † | No Localizer † | |
| LLaVA-1.5-7B | |||||||
| 224 | 101.89% | 101.80% | 98.83% | 101.97% | 101.36% | 101.69% | 101.35% |
| 128 | 100.37% | 99.61% | 96.99% | 100.72% | 98.97% | 100.13% | 98.21% |
| 64 | 97.08% | 95.45% | 94.36% | 97.60% | 93.31% | 96.53% | 92.67% |
| 32 | 92.33% | 89.18% | 90.14% | 92.81% | 86.02% | 91.77% | 85.55% |
| LLaVA-1.5-13B | |||||||
Table 4: Retention of Varest variants at matched budgets. † denotes an inference-time ablation of the joint model Varest. Localization and [REL] pruning provide complementary benefits.
More in this topic
Visual Token Compression
Visual token compression reduces image and video representations to lower inference, memory, storage, or communication costs while retaining task-relevant information.
See all 261 papers