Rethinking Token Reduction for Large Vision-Language Models

Yi Wang; Haofei Zhang; Qihan Huang; Anda Cao; Gongfan Fang; Wei Wang; Xuan Jin; Jie Song

cs.CVMar 23, 2026

Rethinking Token Reduction for Large Vision-Language Models

Authors: Yi Wang, Haofei Zhang, Qihan Huang, Anda Cao, Gongfan Fang, Wei Wang, Xuan Jin, Jie Song,

Organizations: College of Computer Science and Technology, Zhejiang University · State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · National University of Singapore ·

Abstract

Large Vision-Language Models (LVLMs) excel in visual understanding and reasoning, but the excessive visual tokens lead to high inference costs. Although recent token reduction methods mitigate this issue, they mainly target single-turn Visual Question Answering (VQA), leaving the more practical multi-turn VQA (MT-VQA) scenario largely unexplored. MT-VQA introduces additional challenges, as subsequent questions are unknown beforehand and may refer to arbitrary image regions, making existing reduction strategies ineffective. Specifically, current approaches fall into two categories: prompt-dependent methods, which bias toward the initial text prompt and discard information useful for subsequent turns; prompt-agnostic ones, which, though technically applicable to multi-turn settings, rely on heuristic reduction metrics such as attention scores, leading to suboptimal performance. In this paper, we propose a learning-based prompt-agnostic method, termed MetaCompress, overcoming the limitations of heuristic designs. We begin by formulating token reduction as a learnable compression mapping, unifying existing formats such as pruning and merging into a single learning objective. Upon this formulation, we introduce a data-efficient training paradigm capable of learning optimal compression mappings with limited computational costs. Extensive experiments on MT-VQA benchmarks and across multiple LVLM architectures demonstrate that MetaCompress achieves superior efficiency-accuracy trade-offs while maintaining strong generalization across dialogue turns. Our code is available at https://github.com/MArSha1147/MetaCompress.

Figures & tables

Figure 1: (a) Overall pipeline of the compression projection training process. (b) Attention distribution over the [ CLS ] token for retained and all visual tokens. The image tokens are extracted from the last layer of the vision tower of LLaVA-1.5-13b running on VQA-v2 dataset. (c) Attention distribution over the prompt tokens for retained and all visual tokens. The attention scores are averaged to prompt tokens extracted from the first layer of the LLM decoder.

Figure 2: Illustration of our proposed MetaCompress, where module $\mathcal{P}_{\text{meta}}$ generate the compression projection $P$ solely according to the image sequence $X_{\text{IMG}}$ .

Model	Method	MT-VQA-v2				MT-GQA				ConvBench
Model	Method	$Acc_{1}$	$Acc_{2}$	$Acc_{3}$	$Avg$	$Acc_{1}$	$Acc_{2}$	$Acc_{3}$	$Avg$	$S_{1}$	$S_{2}$	$S_{3}$	$Avg$
LLaVA-1.5-7b	Base	76.72	77.51	77.30	77.18	61.76	64.07	65.35	63.73	4.33	5.72	5.55	5.20
	Random	66.36	66.94	66.68	66.66	54.60	57.07	59.31	56.99	3.73	3.99	4.08	3.93
	Sample	67.11	67.52	67.63	67.42	55.06	57.89	59.74	57.56	3.64	4.85	3.81	4.10
	FastV	45.98	48.56	49.65	48.06	40.98	46.71	49.30	45.66	1.56	1.39	3.12	2.02
	PruMerge	69.03	69.93	69.73	69.56	55.26	57.23	60.13	57.54	4.51	3.47	3.47	3.82

Table 1: The comparison of visual token reduction methods on three MT-VQA benchmarks with the reduction rate of 90%. The best and the second-best results are highlighted in bold and underline, respectively.

Panel of Figure 3 — Figure 3: Comparison of average accuracy on MT-GQA with reduction rate from 50% to 95%.

Model	Setting	TTFT	E2ET	Mem.	TFLOPs
LLaVA-1.5-7b	Base	232 ( $\pm$ 5.8)	676 ( $\pm$ 8.0)	26.9	71.4
	Random	98.2 ( $\pm$ 5.7)	487 ( $\pm$ 4.6)	26.2	13.3
	Sample	96.9 ( $\pm$ 5.8)	482 ( $\pm$ 4.9)	26.2	13.3
	FastV	102 ( $\pm$ 4.52)	528 ( $\pm$ 6.0)	26.3	13.5
	PruMerge	107 ( $\pm$ 5.23)	509 ( $\pm$ 4.5)	26.2	13.3
	Ours	97.8 ( $\pm$ 5.41)	480 ( $\pm$ 5.1)	26.1	13.3

Table 2: Efficiency comparison of different token reduction methods. The time to first token (TTFT, ms), end-to-end generation time (E2ET, ms), GPU memory usage (Mem. GB), and TFLOPs are reported on MT-GQA dataset with a reduction rate of 90%.

Settings				MT-GQA
$\mathcal{L}_{\text{pred}}$	$\mathcal{L}_{\text{entropy}}$	$\mathcal{L}_{\text{collapse}}$	$\mathcal{L}_{\text{collapse}}$ + Grad Clip	$Avg$
✓	✗	✗	✗	61.98
✓	✓	✗	✗	62.42
✓	✗	✓	✗	56.34
✓	✗	✗	✓	62.13
✓	✓	✗	✓	62.70

Table 3: Ablation study of training MetaCompress for LLaVA-NeXT-7b using different loss terms on MT-GQA. Gradient clipping is only applied for the ‘

\mathcal{L}_{\text{collapse}}

+ Grad Clip ’ setting.

Figure 4: Sensitivity analysis in training MetaCompress for LLaVA-NeXT-7b with different weights

\alpha_{\text{entropy}}

and

\alpha_{\text{collapse}}

on MT-GQA.

Panel of Figure 5 — Figure 5: (a) Token importance distribution. (b) Attention distribution over prompt tokens. Image tokens are extracted from the last layer of the vision tower of LLaVA-NeXT-7b on VQA-v2 dataset.

Model	Method	MT-VQA-v2				MT-GQA				ConvBench
Model	Method	$Acc_{1}$	$Acc_{2}$	$Acc_{3}$	$Avg$	$Acc_{1}$	$Acc_{2}$	$Acc_{3}$	$Avg$	$S_{1}$	$S_{2}$	$S_{3}$	$Avg$
LLaVA-1.5-7b	Base	76.72	77.51	77.30	77.18	61.76	64.07	65.35	63.73	4.33	5.72	5.55	5.20
	Random	72.72	73.57	73.11	73.13	57.79	61.07	63.09	60.65	4.17	4.73	5.51	4.80
	Sample	72.96	73.71	73.33	73.33	58.48	61.12	62.30	60.63	4.16	5.03	3.81	4.33
	FastV	69.30	69.57	69.41	69.43	54.79	57.65	60.03	57.49	3.99	5.03	3.47	4.16
	PruMerge	72.79	73.90	73.42	73.37	57.89	59.54	61.81	59.75	3.64	4.68	4.68	4.33

Table 4: The comparison of visual token reduction methods on three MT-VQA benchmarks with the reduction rate of 70%. The best and the second-best results are highlighted in bold and underline, respectively.

Setting	MT-GQA
Setting	$Acc_{1}$	$Acc_{2}$	$Acc_{3}$	$Avg$
Base	61.76	64.07	65.35	63.73
Sample	54.60	57.07	59.31	56.99
Spatial	51.05	54.62	56.24	53.97
ToMe	53.64	56.91	57.62	56.06
VisionZip	55.08	57.82	59.89	57.60

Table 5: Comparison results of different token merging methods for LLaVA-1.5-7b.

$\mathcal{L}_{\text{pred}}$	$\mathcal{L}_{\text{entropy}}$	$\mathcal{L}_{\text{collapse}}$	$\mathcal{L}_{\text{collapse}}$ + Grad Clip	LLaVA-1.5-7b	LLaVA-NeXT-7b	XComposer-2.5-7b
✓	✗	✗	✗	56.63	61.98	56.77
✓	✓	✗	✗	57.99	62.42	58.01
✓	✗	✓	✗	52.26	56.34	52.53
✓	✗	✗	✓	57.57	62.13	58.24
✓	✓	✗	✓	58.43	62.70	58.68

Table 6: Additional ablation study of training MetaCompress for various LVLMs using different loss terms on MT-GQA. Gradient clipping is only applied for the ‘

\mathcal{L}_{\text{collapse}}

+ Grad Clip ’ setting.

Settings	LLaVA-1.5-7b	LLaVA-1.5-13b	LLaVA-NeXT-7b	LLaVA-NeXT-13b	XComposer-2.5-7b
MT-VQA-v2	70.65	72.94	75.18	75.26	75.76
MT-GQA $\rightarrow$ MT-VQA-v2	69.06	71.89	73.61	73.25	74.41
MT-GQA	58.43	59.48	62.70	63.12	58.68
MT-VQA-v2 $\rightarrow$ MT-GQA	57.45	58.78	61.43	62.60	58.06

Table 7: Transfer validation experiments. Average accuracy is reported for cross-dataset transfer between MT-VQA-V2 and MT-GQA, all under a 90% token reduction rate.

Metrics	Base	Random	Sample	FastV	Ours
$Acc_{1}$	44.3	26.8	25.5	26.2	28.5
$Acc_{2}$	46.7	25.3	26.4	27.3	27.3
$Acc_{3}$	48.2	30.7	31.3	31.6	34.6
$Avg$	46.4	27.6	27.7	28.4	30.1

Table 8: Transfer results on MT-Video-MME. Average accuracy is reported across different methods.

Panel of Figure 6 — Figure 6: Visualization of the compression projection for LLaVA-1.5-7b on MT-GQA with the compression rate of 90%.

Panel of Figure 7 — Figure 7: Visualization of the compression projection for LLaVA-1.5-13b on MT-GQA with the compression rate of 90%.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference
Feb 21, 2026Aditya Kumar Singh, Hitesh Kandala, Pratik Prabhanjan Brahma +2Efficient VLM Inference Multimodal Token Compression
Advanced Micro Devices, Inc. (AMD)
Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Models
Mar 11, 2026Sihan Cao, Jianwei Zhang, Pengcheng Zheng +7Efficient VLM Inference Visual Token Pruning
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Robotics and Advanced Manufacture, Harbin Institute of Technology, Shenzhen, China · College of Information and Control Engineering, Xi’an University of Architecture and Technology, Xi’an, China
Adaptive Visual Token Reduction for Accelerated Image Understanding
Oct 7, 2026Seyoung Jeong, Jong Pil Yun, Sang Jun LeeEfficient VLM Inference Visual Token Pruning
Jeonbuk National University, Jeonju, Republic of Korea · Korea Institute of Industrial Technology (KITECH), Incheon, Republic of Korea · Chung-Ang University, Seoul, Republic of Korea

Feb 21, 2026J/K move · Enter open · S save

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference

Aditya Kumar Singh, Hitesh Kandala, Pratik Prabhanjan Brahma, Zicheng Liu+1

Advanced Micro Devices, Inc. (AMD)

Efficient VLM Inference Multimodal Token Compression Visual Token Compression Visual Token Pruning Video Token Compression

Open paper AskPDF arXiv

Vision-language models (VLMs) have achieved remarkable multimodal understanding and reasoning capabilities, yet remain computationally expensive due to dense visual tokenization. Existing efficiency approaches either merge redundant visual tokens or drop them progressively in language backbone, often trading accuracy for speed. In this work, we propose DUET-VLM, a versatile plug-and-play dual compression framework that consists of (a) vision-only redundancy aware compression of vision encoder's output into information-preserving tokens, followed by (b) layer-wise, salient text-guided dropping of visual tokens within the language backbone to progressively prune less informative tokens. This coordinated token management enables aggressive compression while retaining critical semantics. On LLaVA-1.5-7B, our approach maintains over 99% of baseline accuracy with 67% fewer tokens, and still retains >97% even at 89% reduction. With this dual-stage compression during training, it achieves 99.7% accuracy at 67% and 97.6% at 89%, surpassing prior SoTA visual token reduction methods across multiple benchmarks. When integrated into Video-LLaVA-7B, it even surpasses the baseline -- achieving >100% accuracy with a substantial 53.1% token reduction and retaining 97.6% accuracy under an extreme 93.4% setting. These results highlight end-to-end training with DUET-VLM, enabling robust adaptation to reduced visual (image/video) input without sacrificing accuracy, producing compact yet semantically rich representations within the same computational budget. Our code is available at https://github.com/AMD-AGI/DUET-VLM.

Abstract

Efficient VLM Inference

Explore similar work

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference

Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Models

Adaptive Visual Token Reduction for Accelerated Image Understanding