Bottlenecked semantic compression by representation reconstruction

Xiaoya Tang; Shilong Liu; Daniel Smolyak; Xiwen Li; Quynh Nguyen; Tolga Tasdizen

cs.CVOct 12, 2026

Bottlenecked semantic compression by representation reconstruction

Authors: Xiaoya Tang, Shilong Liu, Daniel Smolyak, Xiwen Li, Quynh Nguyen, Tolga Tasdizen

Organizations: Scientific Computing and Imaging Institute University of Utah · Princeton AI Lab Princeton University · National Institutes of Health

Abstract

Vision Transformers incur high computational costs when processing long visual token sequences. Existing reduction methods typically rank tokens independently and discard those deemed unimportant or redundant. We propose a reconstruction-guided framework for visual token condensation (ReConV), which summarizes the complete visual sequence with a compact Summary Bank (SMB) and retains tokens whose information is not adequately captured by the SMB, as indicated by their reconstruction residuals. At each compression layer, a lightweight position-conditioned decoder reconstructs the current tokens, leaving shared information compressed in the SMB while explicitly preserving poorly reconstructed details. Experiments on image classification and multimodal understanding demonstrate favorable performance-efficiency trade-offs: ReConV surpasses the uncompressed baseline with 48% fewer GMACs on ImageNet and enables reusable image-level representations for efficient multi-turn inference. Code is available at TokenDrop.

Figures & tables

Method	No attention-layer / head dependence	Prompt independence	Reduction before the LLM	Preserves uncaptured information	Joint token redundancy / coverage
Decoder-attention ranking	✗	✗	✗	✗	✗
Encoder-attention ranking	✗	✓	✓	✗	✗
Other learned scoring / gating	✓	✓	✓	✗	✗
Similarity-based clustering / merging	✓	✓	✓	✗	✓
Summary Bank + residual(Ours)	✓	✓	✓	✓	✓

Table 1: Comparison of representative visual token reduction paradigms. A ✓ denotes explicit design support, whereas an ✗ denotes no explicit guarantee. No attention-layer/head dependence excludes reliance on specific attention layers or heads; prompt independence excludes textual conditioning; reduction before the LLM means compression precedes LLM processing; preserving uncaptured information retains information missing from the compressed representation; and joint token information coverage evaluates tokens relative to the selected set rather than independently.

Figure 1: Illustration of visual token pruning. Prompt-independent saliency emphasizes irrelevant regions, while prompt-conditioned pruning can still discard task-relevant details, especially at high pruning ratios. ReConV instead preserves information through SMB summaries and residual tokens.

Figure 2: Overview of ReConV, the proposed reconstruction-guided token pruning framework. Given $N$ token embeddings from a vision encoder or transformer layer, each compressive layer uses a small set of learned summary bank (SMB) tokens to summarize the input via cross-attention, reconstructs the original tokens using the SMB representation and positional information, and retains the top- $k$ tokens with the largest reconstruction residuals (approximately $N\cdot r$ , where $r$ is the keep ratio). Tokens that are well reconstructed are dropped as redundant, while the retained tokens are passed to the next layer. For classification, the model uses the CLS token when available, or pooled SMB features otherwise; for other tasks, it uses the retained visual tokens with SMBs.

Panel of Figure 3 — Figure 3: Patch-to-SMB assignments for tokens entering the first, second, and third compression layers of our A1A3A5 classification model, which use keep ratios of $0.75$ , $0.25$ , and $0.0$ , respectively. The model’s performance is reported in Table 2 . Green numbers identify the SMB tokens assigned to reconstruct the corresponding patches, and patches assigned to the same SMB are masked with the same color. Unnumbered patches have been dropped by preceding layers. Examples are randomly selected from the ImageNet validation set.

Model	GMACs	Accuracy
DeiT-S(Base)	4.61 ( 100% )	79.8 ( 100% )
A-ViT Yin et al. (2022) (CVPR’22)	$3.6\,(\downarrow 22\%)$	$78.6\,(\text{98.5\%})$
STViT Chang et al. (2023) (CVPR’23)	$3.16\,(\downarrow 31\%)$	$80.6\,(\text{101.0\%})$
EViT Liang et al. (2022) (ICLR’22)	$3.0\,(\downarrow 35\%)$	$79.5\,(\text{99.6\%})$
Evo-ViT Xu et al. (2022) (AAAI’22)	$3.0\,(\downarrow 35\%)$	$79.4\,(\text{99.5\%})$
Dynamic ViT Rao et al. (2021) (NeurIPS’21)	$2.9\,(\downarrow 37\%)$	$79.3\,(\text{99.4\%})$

Table 2: Comparison of computational cost and accuracy on ImageNet. The percentage in GMACs indicates the reduction computation relative to the backbone, and the percentage in accuracy indicates the retained performance relative to the backbone accuracy. The configurations “0.75,0.25,0” denote the keep ratios of the three compressive layers, corresponding to

\rho_{1},\rho_{2},\rho_{3}

in Section 3.3 . Our distill indicates that a DeiT-S teacher trained from scratch is used for distillation.

Panel of Figure 4 — Figure 4: Examples from the MT-VQAv2 test split, where the number of questions associated with each image varies across samples. In the multi-turn setting, each image is encoded and optionally compressed only once, and the resulting visual embeddings are reused across all associated questions.

Method	MT-VQA-v2						ConvBench
Method	Yes/No	Number	Other	Overall	Amort. (ms/q) $\downarrow$	Speedup ( $\times$ ) $\uparrow$	$S_{1}$	$S_{2}$	$S_{3}$	Average	Mean Total Prefill (s) $\downarrow$	Speedup ( $\times$ ) $\uparrow$
Uncompressed	93.28	62.22	72.22	79.99	314.50	1.00	3.47	4.33	4.68	4.16	361.05	1.00
Random Pruning	87.19	47.43	59.97	69.76	114.91	2.74	3.29	3.29	2.60	3.06	242.06	1.49
Spatial Pooling	80.41	38.60	50.58	61.51	114.32	2.75	3.12	3.12	2.77	3.00	240.92	1.50
FastV[ 3 ]	80.56	40.75	54.15	63.51	192.40	1.63	3.81	3.64	3.47	3.64	739.47	0.49
PDrop[ 28 ]	75.71	30.56	42.05	54.61	119.30	2.64	2.60	2.95	2.60	2.72	237.76	1.52

Table 3: Comparison on MT-VQA-v2 and ConvBench at a 95% visual token reduction rate. Amort. denotes latency per question with image encoding amortized across questions, while Mean Total Prefill denotes total prefill latency averaged across conversations.

Panel of Figure 5 — Figure 5: ImageNet1K top-1 accuracy percentage vs computation cost (GMACs) tradeoff. Left: Models with 1-2 compressive layers with varying number of SMBs and layer placement. Right: Models with 3 compressive layers (A1A3A5) with varying number of SMBs and keep ratios.

Ablation	Inter-layer repr.	LLM input	Position reinjection	Total tokens	# Retained tokens	# SMB tokens	# Compression layers	MME-P Score	MME-C Score
LLaVA-1.5-13B (Base)	Uncompressed (576 tokens)							1531	295
Representation propagation	Original	Original	✓	127	119	8	2	1242	273
	Original	Residual	✓	127	119	8	2	1370	285
	Residual	Original	✓	127	119	8	2	1405	305
	Residual	Residual	✓	127	119	8	2	1457	291
Positional information	Residual	Residual	✗	127	119	8	2	1337	285

Table 4: Ablation study of the compression design on multimodal understanding. Inter-layer repr. and LLM input specify whether original or residual tokens are propagated between compression layers and supplied to the LLM, respectively.

K

denotes the total token budget, including retained and SMB tokens. MME-P and MME-C denote the total perception and cognition scores.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Heads	Dim	Layers	Teacher	SMB	Configuration	Keep ratios	GMACs	Accuracy
Vanilla	6	384	12	–	–	–	–	4.61	80.61
DeiT-S	6	384	12	regnet-y-160	–	–	–	4.61	81.88
Comp-R	6	384	12	Vanilla	16	A1A3A5	0.75,0.25,0.25	2.43	79.61
Comp-F	6	384	12	Vanilla	16	A1A3A5	0.75,0.25,0.25	2.43	79.64
Comp-R	6	384	12	Vanilla	16	A1A3A5	0.75,0.25,0.00	2.33	79.75
Comp-F	6	384	12	Vanilla	32	A1A3A5	0.75,0.25,0.25	2.88	80.44

Appendix

Table 5: ImageNet1k top-1 accuracy and computational cost of the proposed compressive model under various settings. comp-f and comp-r refer to our model using compressive layers that output full vs residual tokens, respectively.

Panel of Figure 6 — Figure 6: Patch-to-SMB assignments in our single-layer A5 classification model with a keep ratio of $0.0$ and our three-layer A1A3A5 classification model with keep ratios of $0.75$ , $0.25$ , and $0.0$ . From left to right, the figure shows the patch-to-SMB assignments and corresponding color-coded masks at the single compression layer of the A5 model, followed by the patch-to-SMB assignments for the tokens entering the first, second, and third compression layers of the A1A3A5 model. For example, the visualization at the first layer therefore includes all input patch tokens; this layer subsequently drops $25\%$ of them, and the retained tokens enter the second compression layer. Green numbers identify the SMB tokens assigned to reconstruct the corresponding patches, with patches assigned to the same SMB sharing the same color. Patches without a number have been dropped by preceding compression layers. The examples are randomly selected from the ImageNet validation set.

Model	Ablation	Accuracy
A1A3A5-SMB16- $0.75,0.25,0.0$	–	79.92
A1A3A5-SMB16- $0.75,0.25,0.0$	return residual tokens	79.93
A1A3A5-SMB16- $0.75,0.25,0.0$	no decoder	75.95
A1A3A5-SMB16- $0.75,0.25,0.0$	random token dropping	79.21
A1A3A5-SMB16- $0.75,0.25,0.0$	decoder without positional input	79.91
A1A3A5-SMB16- $0.75,0.25,0.0$	late SMB injection	79.70

Appendix

Table 6: Ablation studies on model components and architectural design choices.

Method	VQA v2	SQA IMG	TextVQA	POPE	MME	MMB	MMB CN	Avg.
Vanilla 576 Tokens
LLaVA-1.5-13B	80.0	71.6	61.3	86.0	1531	67.7	63.5	100%
Retain 128 Tokens
ToMe(ICLR’23) [ 1 ]	-	59.6	49.1	62.8	1343	53.3	-	79.6%
FastV 4	75.3	74.2	58.6	75.5	1460	66.1	62.3	96.0%
PDrop 28	78.2	73.3	60.2	83.6	1489	67.5	62.8	98.4%

Appendix

Table 7: Comparison of token retention methods on LLaVA-1.5-13B across varying token budgets. Raw scores are reported in the first row and average percentages to the baseline in the final column. Results for SparseVLM and ToMe are sourced from Zhang et al. (2024) . Comparisons are best interpreted via relative percentages. The baseline scores 76.4 and 72.4 on the dev splits of MMB and MMB CN , respectively

Panel of Figure 7 — Figure 7: Samples are randomly selected from the MME evaluation set, covering five subtasks from left to right: artwork, existence, landmark, OCR, and scene. The figure visualizes the 64-token setting of our model, which contains 56 kept tokens and 8 SMBs. From top to bottom, the panels show: the question-answer pair, the input image, SMB allocation at the first compression layer, SMB allocation at the second compression layer after the first token-dropping step, and the final SMB allocation after the second token-dropping step. The color from light to dark yellow indicates SMB indices from 0 to 8. The letters on each patch indicate the assigned SMB index, while patches without letters have already been dropped.

Method	Tokens	POPE Accuracy			Prefill Latency	Speedup
Method	Tokens	rand	pop	adv	/Sample(ms) $\downarrow$	(Prefill) $\uparrow$
Baseline	576	87.1	86.2	84.5	112.25	1.00 $\times$
ReConV(Ours)	127	83.3	82.8	80.4	49.63	2.26 $\times$
ReConV(Ours)	64	84.5	82.9	80.3	37.84	2.97 $\times$
ReConV(Ours)	32	84.8	82.5	78.9	36.55	3.07 $\times$

Appendix

Table 8: Efficiency analysis on POPE. We report prefill latency per sample, measured on an A800 GPU (40GB). Speedup is computed relative to the Baseline.

Explore similar work

CoViST: Visual Token Compression via Composable States
Sep 27, 2026Qi Zhang, Xiandong Meng, Ronggang Wang +1Vision-Language Models Efficient VLM Inference
Peng Cheng Laboratory Shenzhen, China · Peking University Shenzhen Graduate School Peking University Beijing, China
VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression
Jul 14, 2026Yupeng Zheng, Kai Zou, Bin Liu +1Efficient VLM Inference Large Vision-Language Models
Anhui Province Key Laboratory of Digital Security, University of Science and Technology of China Hefei, China
DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference
Feb 21, 2026Aditya Kumar Singh, Hitesh Kandala, Pratik Prabhanjan Brahma +2Efficient VLM Inference Multimodal Token Compression
Advanced Micro Devices, Inc. (AMD)

Sep 27, 2026J/K move · Enter open · S save

CoViST: Visual Token Compression via Composable States

Qi Zhang, Xiandong Meng, Ronggang Wang, Siwei Ma

Peng Cheng Laboratory Shenzhen, China · Peking University Shenzhen Graduate School Peking University Beijing, China

Vision-Language Models Efficient VLM Inference Visual Token Compression

Open paper AskPDF arXiv

Visual token compression lowers the inference cost of vision--language models by representing images with fewer tokens. However, most existing methods compress visual tokens to a reduced set, leaving the amount of visual evidence represented by each token and its original spatial context implicit. Therefore, the compressed representation does not explicitly encode how much visual information each representative carries or where it lies in the original image. This limitation arises even after a single reduction and becomes more pronounced when compression is repeated across decoder layers. To address this issue, we propose CoViST, a training-free framework that represents a compressed image as a composable visual state. Specifically, the state combines representative features with original positions, effective contribution weights, and reusable selection metadata. CoViST constructs this state through coverage-guided selection and conservation-based contribution composition, and explicitly incorporates its contribution and positional information into decoder attention. Each component of the state retains its interpretation under successive reductions, enabling the same formulation to support both fixed compression before prefill and progressive compression within the decoder. Experimental results on seven LLaVA-1.5-7B benchmarks show that CoViST-Fixed retains 99.9%, 99.5%, and 98.1% of uncompressed performance at 192, 128, and 64 tokens, respectively, and CoViST-Pro retains 99.8%, 99.9%, and 99.1% at the corresponding layer-average budgets, outperforming state-of-the-art methods under their respective budget settings. Code will be released publicly.

Abstract

Explore similar work

CoViST: Visual Token Compression via Composable States

VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference