Foveated Compression: Selective High-Resolution Preservation for Token-Efficient VLMs

Donghyun Han; Jangho Park; Yuseok Bae

cs.CVOct 6, 2026

Foveated Compression: Selective High-Resolution Preservation for Token-Efficient VLMs

Authors: Donghyun Han, Jangho Park, Yuseok Bae

Organizations: ETRI, South Korea · Kyung Hee University, South Korea

Abstract

Visual tokens are a major source of inference cost in vision-language models, yet simple image downsampling remains a surprisingly strong compression baseline. This raises a complementary question: under a fixed token budget, where should visual fidelity be preserved? We introduce Foveated Compression, which encodes a full-resolution image once and represents it with a mixture of native- and compressed-resolution visual tokens. A behaviorally self-distilled Foveated Merger compresses local visual tokens while preserving compatibility with their native counterparts, and a lightweight Foveated Selector chooses one of nine spatial cells to retain at native resolution using exhaustive budget-matched intervention supervision. At 11.11% visual tokens, uniform Foveated Compression shows no significant paired difference from iso-token downsampling. At 20.99%, the learned selector significantly outperforms random and fixed allocation, but remains below strong whole-image resizing, showing that localized fidelity is not universally preferable. A budget-matched region-choice oracle reaches 82.73 macro accuracy versus 69.61 for the learned selector, revealing substantial headroom within the same spatial action space. Matched probing further shows that signals predicting when compression breaks the answer are substantially more accessible after language-model computation than to the lightweight prefill-free selector. These results expose complementary bottlenecks in region selection and compressed-region fidelity.

Figures & tables

Figure 1 : Foveated Compression preserves one selected region at native visual fidelity while keeping the rest of the image compressed within a single full-resolution encoding. Unlike whole-image downsampling or selective cropping, it maintains global coverage and enables native-token restoration without additional high-resolution visual encoding.

Figure 2 : Overview of Foveated Compression. The image is encoded once by the frozen vision backbone, the Foveated Merger produces a globally compressed representation, and the Foveated Selector restores one selected spatial cell with native visual tokens through in-place replacement.

Panel of Figure 3 — Figure 3 : High-resolution diagnostics. (a) Native-correct retention under uniform compression. (b) Spatial informativeness and winner-region adjacency. (c) Performance headroom from the budget-matched $K{=}1$ region-choice oracle.

Method	Tokens	GQA	MMB-E	MMB-C	MME	POPE	MMStar	OCR	ChartQA	Macro
Native	100.00%	60.97	86.86	85.61	2356.7	87.74	60.93	80.20	84.28	78.85
Uniform Foveated Compression	11.11%	58.28	82.51	82.01	2130.5	84.34	50.20	49.30	34.24	64.62
Iso-Token Downsample	11.11%	55.88	82.77	81.75	2240.8	82.69	51.07	57.30	27.44	64.86
Random $K{=}1$	20.99%	58.40	84.18	82.86	2190.7	84.34	52.33	55.10	48.36	67.98
Fixed $K{=}1$	20.99%	58.90	83.69	82.70	2170.3	84.54	51.87	53.70	54.92	68.48
Foveated Selector	20.99%	58.24	84.59	82.67	2186.7	84.92	52.13	58.40	57.84	69.61

Table 1: Main results on 37,610 examples across eight benchmarks. Bold and underlined values denote the best and second-best methods within the approximately 20% fixed-budget comparison group. MME is divided by 28 only for the macro average.

Variant	Tokens	GQA	MMB-E	MMB-C	MME	POPE	MMStar	OCR	ChartQA	Macro
Mean-Pooling Compression	11.11%	51.76	79.65	78.33	2013.4	80.47	44.73	41.40	24.12	59.05
Behaviorally Trained Merger	11.11%	58.28	82.51	82.01	2130.5	84.34	50.20	49.30	34.24	64.62
Single-Winner Supervision	20.99%	57.47	83.51	81.75	2203.5	83.80	52.47	56.20	54.40	68.54
Tie-Aware Supervision	20.99%	58.38	85.08	83.04	2189.5	84.86	52.40	59.20	57.04	69.77
Question Content Removed	20.99%	58.42	84.59	82.65	2205.7	85.01	53.00	58.90	57.08	69.80
Full Selector	20.99%	58.38	85.08	83.04	2189.5	84.86	52.40	59.20	57.04	69.77

Table 2 : Per-benchmark ablations. The merger block uses the 11.11% uniform setting; selector ablations use a matched full-selector rerun (69.77 macro), distinct from the 69.61 main checkpoint. MME is divided by 28 only for the macro average. The Tie-Aware Supervision and Full Selector rows report the same run, listed once in each ablation block.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Merger	Selector
Trainable parameters	32.77M	10.8M
Learning rate	$1\times 10^{-4}$	$2\times 10^{-4}$
Epochs	1	4
Warmup steps	200	100
Optimization steps	2,378	–
Gradient accumulation	4	–

Appendix

Table A1: Main training settings for the Foveated Merger and Foveated Selector. Settings not shared across the two modules are marked with “–”.

Condition	Visual-token ratio
Native	100.00%
Uniform Foveated Compression	11.11%
Iso-Token Downsample	11.11%
Random / Fixed / Learned $K{=}1$	20.99%
Matched-Budget Downsample	19.82%
$2\times$ Downsample	24.57%

Appendix

Table A2: Visual-token budgets relative to the 252-aligned native input.

Statistic	Value
Total selector labels	18,048
Spatially informative labels	7,678
Informative fraction	42.54%
Mean winner-set size	3.66
Random winner membership	40.65%
Best training-only fixed cell	45.40%

Appendix

Table A3: Statistics of the intervention-derived selector supervision. Winner membership measures whether the selected cell belongs to the set of equally optimal

K{=}1

actions.

Diagnostic	Value
Evaluation examples	37,610
Overall native-correct retention	86.1%
Lowest retention (ChartQA)	38.8%
Highest retention (POPE)	94.3%
All-nine-tie examples	78.5%
Spatially informative examples	21.5%

Appendix

Table A4: Aggregate resolution-sensitivity and spatial-choice diagnostics.

Comparison	$\Delta$ Macro	95% CI
Uniform Foveated Compression $-$ Iso-Token Downsample	$-0.24$	$[-0.586,\;0.521]$
Foveated Selector $-$ Native	$-9.238$	$[-9.867,\;-8.625]$
Foveated Selector $-$ Random $K{=}1$	$+1.622$	$[1.085,\;2.161]$
Foveated Selector $-$ Fixed $K{=}1$	$+1.131$	$[0.610,\;1.632]$
Foveated Selector $-$ Matched-Budget Downsample	$-1.378$	$[-2.062,\;-0.707]$
$2\times$ Downsample $-$ Foveated Selector	$+2.802$	$[2.164,\;3.438]$

Appendix

Table A5: Paired bootstrap comparisons. Positive values favor the first method named in each comparison. The oracle is a non-deployable test-time upper bound.

Signal	AUC	95% CI
Prefill-free raw signal	0.5239	$[0.4796,\;0.5700]$
First-token entropy	0.7651	$[0.7347,\;0.7937]$
Prefill-free OOF probe	0.5476	$[0.5012,\;0.5929]$
Decoder-side OOF probe	0.8072	$[0.7738,\;0.8391]$

Appendix

Table A6: Matched AUC comparison for predicting compression-induced answer failure.

Panel of Figure 12 — Figure A1 : Additional qualitative examples across three outcome regimes. Each column contains examples of scene-text reading, object existence, and compositional reasoning. (a) Compression-robust: downsampling, the learned selector, and the $K{=}1$ oracle all succeed. (b) Selection-rescued: downsampling fails, while the learned selector preserves a region that recovers the correct answer. (c) Selection-miss: both downsampling and the learned selector fail, but another $K{=}1$ oracle action succeeds. Oracle regions are shown only as a diagnostic upper bound and are not available to the deployable selector.

Explore similar work

AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inference
May 28, 2026Yilin Feng, Ahmed Burak Gulhan, Mahmut Taylan KandemirVisual Token Pruning Large Language Model Compression
The Pennsylvania State University
Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression
Aug 3, 2026Long Qian, Jiaqi Wei, Bingke Zhu +2Large Language Model Compression Learned Image Compression
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Engineering, Cardiff University, Cardiff, United Kingdom · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China +1
OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning
May 28, 2026Geng Li, Guohao Chen, Ting Chen +6Visual Token Pruning Long Visual-Token Sequences
1Nanyang Technological University (NTU)

May 28, 2026, cs.LGJ/K move · Enter open · S save

AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inference

Yilin Feng, Ahmed Burak Gulhan, Mahmut Taylan Kandemir

The Pennsylvania State University

Visual Token Pruning Large Language Model Compression Structured Pruning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) process thousands of visual tokens per image alongside comparatively few text tokens, yet existing compression methods treat both modalities uniformly. We observe that the two modalities have fundamentally different properties: vision tokens are spatially redundant and dominate prefill, while text tokens are causally dependent and accumulate during decoding. Based on this asymmetry, we propose and empirically evaluate AsymVLM, which applies aggressive pruning to vision tokens before prefill using a learned importance scorer with per-sample adaptive budgeting, and temporal threshold-based eviction to text tokens only when they exceed a fixed budget. Our experiments indicate that AsymVLM achieves the highest FLOPs savings (up to 54%) among state-of-the-art methods while outperforming existing approaches by 2--3% on document and chart understanding tasks where visual information is spatially localized and query-specific, and maintaining competitive accuracy on holistic benchmarks. In text-dominated scenarios, our eviction strategy substantially outperforms standard LLM cache compression methods by adapting to the short-context nature of VLM.

Abstract

Explore similar work

AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inference

Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning