DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference

Aditya Kumar Singh; Hitesh Kandala; Pratik Prabhanjan Brahma; Zicheng Liu; Emad Barsoum

cs.CVFeb 21, 2026

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference

Authors: Aditya Kumar Singh, Hitesh Kandala, Pratik Prabhanjan Brahma, Zicheng Liu, Emad Barsoum

Organizations: Advanced Micro Devices, Inc. (AMD)

Abstract

Vision-language models (VLMs) have achieved remarkable multimodal understanding and reasoning capabilities, yet remain computationally expensive due to dense visual tokenization. Existing efficiency approaches either merge redundant visual tokens or drop them progressively in language backbone, often trading accuracy for speed. In this work, we propose DUET-VLM, a versatile plug-and-play dual compression framework that consists of (a) vision-only redundancy aware compression of vision encoder's output into information-preserving tokens, followed by (b) layer-wise, salient text-guided dropping of visual tokens within the language backbone to progressively prune less informative tokens. This coordinated token management enables aggressive compression while retaining critical semantics. On LLaVA-1.5-7B, our approach maintains over 99% of baseline accuracy with 67% fewer tokens, and still retains >97% even at 89% reduction. With this dual-stage compression during training, it achieves 99.7% accuracy at 67% and 97.6% at 89%, surpassing prior SoTA visual token reduction methods across multiple benchmarks. When integrated into Video-LLaVA-7B, it even surpasses the baseline -- achieving >100% accuracy with a substantial 53.1% token reduction and retaining 97.6% accuracy under an extreme 93.4% setting. These results highlight end-to-end training with DUET-VLM, enabling robust adaptation to reduced visual (image/video) input without sacrificing accuracy, producing compact yet semantically rich representations within the same computational budget. Our code is available at https://github.com/AMD-AGI/DUET-VLM.

Figures & tables

Panel of Figure 1 — Figure 1: Efficiency and accuracy comparison of DUET-VLM. (a) shows the average inference-only accuracy of VisionZip, PyramidDrop, and DUET-VLMacross different token budgets, compared to the full 576-token LLaVA-1.5-7B baseline on four benchmarks. (b) demonstrates that trained DUET-VLMmodel achieves a 31% reduction in training time while incurring less than a 1% drop in accuracy relative to the LLaVA-1.5-7B baseline.

Figure 2: Overview of the proposed pipeline. An input image is first encoded into $N$ visual tokens by the Vision Encoder . (A) Based on the V2V self-attention map $A_{v2v}^{.,i}\coloneqq 1/N\sum_{j}A_{v2v}^{j,i}$ , we select the most influential Top- $k_{1}$ dominant tokens , while the remaining tokens, $\mathbf{X}_{\text{res}}$ , are merged into $k_{2}$ contextual tokens via attention-guided clustering with a fixed cluster width $w$ , to reduce redundancy. (B) The resulting reduced visual tokens, $\mathbf{X}_{\text{out}}$ , are fed into a language backbone after projecting it through an MLP Adapter , where salient text tokens, $\mathcal{S}$ , further prune visual tokens based on cross-attention scores $A_{t2v}$ at certain selected layers called stages .

\mathcal{C}\coloneqq\text{TopK}(\{s_{i}\}_{i\in\mathcal{R}},k_{2})

(1)

Algorithm 1 Attention guided Dominant selection + Residual Clustering

Method	POPE	SQA I	VQA T	MME	GQA	Avg
Total 576 Tokens (100.0%)
LLaVA-1.5-7B	85.9 (100.0%)	69.5 (100.0%)	58.2 (100.0%)	1862 (100.0%)	61.9 (100.0%)	100.0%
Avg 192 Tokens ( ↓ 66.7% )
FastV [ 5 ] [ECCV’24]	64.8 (75.4%)	67.3 (96.8%)	52.5 (90.2%)	1612 (86.6%)	52.7 (85.1%)	86.8%
HiRED [ 1 ] [AAAI’25]	82.8 (96.4%)	68.4 (98.4%)	47.4 (81.4%)	1737 (93.3%)	58.7 (94.8%)	92.9%
FitPrune [ 31 ] [AAAI’25]	83.4 (97.1%)	67.8 (97.6%)	5 7.4 ( 9 8.6%)	1831 (98.3%)	60.4 (97.6%)	9 7.8%

Table 1: Comparison of inference-only methods on LLaVA-1.5-7B. We report results across five benchmarks under different average token budgets, corresponding to 67%, 78%, and 89% token reduction. DUET-VLM (C) achieves the highest average performance across all settings, maintaining over 99% of the baseline accuracy while using significantly fewer tokens.

Method	Speedup	POPE	SQA I	VQA T	MME	GQA	Avg
Dynamic Tokens
Qwen-2.5-VL-7B [ 3 ]	1.00 $\times$	87.8 (100.0%)	86.2 (100.0%)	72.3 (100.0%)	2264 (100.0%)	58.3 (100.0%)	100.0%
Avg 640 Tokens
VisionZip	1.3x	87.8 (100.0%)	86.2 (100.0%)	71.2 (98.5%)	2267 (100.1%)	58.3 (100.0%)	99.7%
DUET-VLM (C)	1.3x	88.3 (100.6%)	85.8 (99.5%)	71.5 (98.9%)	2264 (100.0%)	58.5 (100.3%)	99.9%
Avg 320 Tokens

Table 2: Comparison of inference-only methods on Qwen-2.5-VL-7B . We report results across five benchmarks under different average token budgets. DUET-VLM (C) achieves the highest average performance across all settings, maintaining over 98% of the baseline accuracy while using significantly fewer tokens.

Method	POPE	SQA I	VQA T	MME	GQA	Avg
Total 576 Tokens
LLaVA-1.5-7B [ 16 ]	85.9 (100.0%)	69.5 (100.0%)	58.2 (100.0%)	1862.0 (100.0%)	61.9 (100.0%)	100.0%
Avg 192 Tokens ( ↓ 66.7% )
VisionZip	84.9 (98.8%)	68.2 (98.1%)	57.8 (99.3%)	1834 (98.5%)	6 0.1 ( 9 7.1%)	9 8.4%
DUET-VLM (C)	85.5 (99.5%)	6 9.6 ( 1 00.1%)	5 8.2 ( 1 00.0%)	1 826 ( 9 8.1%)	62.4 (100.8%)	99.7%
DUET-VLM (C+all)	8 5.4 ( 9 9.4%)	6 9.6 ( 1 00.1%)	57.7 (99.1%)	1790 (96.1%)	57.6 (93.1%)	97.6%

Table 3: Comparison of trained methods on LLaVA-1.5-7B. We report scores across multiple benchmarks under different visual token budgets. DUET-VLM variants (C), (C+all), and (C+S) defined in Sec. 4.1 consistently outperform or match prior compression approaches such as PyramidDrop and VisionZip, even with reductions of up to 88.9% in visual tokens. These results highlight the effectiveness of training of our dual-stage token selection strategy in preserving baseline accuracy under aggressive compression.

Method	TGIF	MSVD	MSRVTT	Avg
Upper Bound, 2048 Tokens (100%)
Video-LLaVA	47.1	69.8	56.7	100.0%
Avg 960 Tokens ( ↓ 53.12% )
PyramidDrop	46.9	70.0	58.0	100.7%
DUET-VLM (C)	48.9	70.1	55.6	100.8%
Avg 136 Tokens ( ↓ 93.4% )

Table 4: Comparison of different methods on Video-LLaVA-7B. We evaluate performance across three benchmarks to demonstrate that DUET-VLM preserves accuracy even under substantial token compression.

Method	Tokens	Speedup	Avg
LLaVA-1.5-7B	576 (100%)	1x	100.0%
VisionZip	192 (33%)	1.1x	97.7%
PyramidDrop	192 (33%)	1x	96.4%
DUET-VLM (C)	192 (33%)	1.1x	99.0%
DUET-VLM (C)	64 (11%)	1.2x	95.4%

Table 5: Inference-only speedup comparison on LLaVA-1.5-7B. DUET-VLM (C) achieves a strong accuracy-latency trade-off under reduced token budgets.

Method	Tokens	Train Time $\downarrow$	Avg
LLaVA-1.5-7B	576	0%	100.0%
DUET-VLM (C)	192	26%	99.7%
DUET-VLM (C)	128	31%	99.1%
DUET-VLM (C)	64	36%	95.4%

Table 6: Training time comparison on LLaVA-1.5-7B. DUET-VLM (C) offers favorable accuracy–efficiency trade-offs with over 30% reduction in training time while still maintaining over 99% accuracy.

Figure 3: Ablation on varying cluster width of DUET-VLM (C) on VQA T benchmark on LLaVA-1.5-7B for different token budgets.

M	POPE	SQA I	VQA T	MME	GQA	Seed I	Avg
Total 576 Tokens (100.0%)
Base	85.9	69.5	58.2	1862	61.9	66.1	100.0%
Avg 192 Tokens ( ↓ 66.7% )
VZ	85.3	68.9	57.3	1783	59.3	59.9	96.5%
VZ (C)	85.8	68.6	57.6	1743	59.0	63.4	97.1% 97.1% 97.1% 97.1% 97.1% 97.1% 97.1% 97.1% 97.1% 97.1% 97.1% 97.1% 97.1% 97.1% 97.1% 97.1% 97.1% 97.1%
DV (V)	85.6	68.2	55.0	1802	58.8	64.6	97.0%

Table 7: Comparison showing benefts of our local cluster aggregation on LLaVA-1.5-7B across 6 benchmarks. We report numbers for different methods (M): LLaVA-1.5-7B (Base), Vanilla (V), VisionZip (VZ), VisionZip with our proposed clustering (VZ (C)), and DUET-VLM (Vanilla) (DV (V)) variants defined in Sec. 4.1

Figure 4: Attention heatmap of salient text tokens attending to visual tokens at the 9 th layer of the language backbone in DUET-VLM (C+S).

M	POPE	SQA I	VQA T	MME	GQA	Seed I	Avg
Total 576 Tokens (100.0%)
Base	85.9	69.5	58.2	1862	61.9	66.1	100.0%
Avg 192 Tokens ( ↓ 66.7% )
(C)	86.5	69.6	57.7	1817	60.3	65.1	98.9%
(C+all)	86.4	68.2	57.8	1813	60.2	65.1	98.5%
(C+S)	86.4	68.2	57.7	1823	60.2	65.1	98.6%

Table 8: Comparison of DUET-VLM on LLaVA-1.5-7B under different text token selection schemes for language model-side pruning. We evaluate different Methods (M): LLaVA-1.5-7B (Base) and our clustering-based variants defined in Sec. 4.1 across different benchmarks. DUET-VLM maintains strong accuracy even with up to 88.9% fewer tokens highlighting the effectiveness of our dual-stage visual token compression.

M	POPE	SQA I	VQA T	MME	GQA	Seed I	Avg
Upper Bound, 2880 Tokens (100.0%)
Base	86.4	70.2	61.3	1842	64.2	70.2	100.0%
Avg 640 Tokens ( ↓ 77.8% )
(C)	87.2	68.4	59.5	1839	61.8	68.5	98.2%
(C+all)	87.2	68.5	59.4	1840	61.7	68.5	98.2%
(C+S)	87.2	68.5	59.5	1839	61.6	68.5	98.2%

Table 9: Comparison of DUET-VLM on LLaVA-NeXT-7B under different text token selection schemes for language model-side pruning. We evaluate different methods (M): LLaVA-NeXT-7B (Base), and our clustering-based variants defined in Sec. 4.1 . DUET-VLM maintains strong accuracy

>

91% even at 94.4% reduction in tokens.

Figure 5: Performance of various dropping configurations in the language backbone on TextVQA using all text tokens. Red crosses indicate configurations in which all visual tokens are removed at some layer prior to the final layer.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Attention heatmap of salient text tokens attending to visual tokens at the 24 th layer of the language backbone in DUET-VLM (C+S).

Figure 7: Plot of TextVQA performance across varying

k_{1}

for different cluster widths, under the constraint

k_{1}+k_{2}=

307 needed to meet a token budget of 192 using DUET-VLM (C) on LLaVA-1.5-7B.

Figure 8: Plot of TextVQA performance across varying

k_{1}

for different cluster widths, under the constraint

k_{1}+k_{2}=

205 needed to meet a token budget of 128 using DUET-VLM (C) on LLaVA-1.5-7B.

M	POPE	SQA I	VQA T	MME	GQA	Seed I	Avg
Upper Bound, 2880 Tokens (100%)
V	86.4	70.2	61.3	1842	64.2	70.2	100.0%
Avg 640 Tokens ( ↓ 77.8% )
VZ	86.3	68.1	60.2	1787	61.3	66.7	97.1%
VZ (C)	86.4	67.8	60.1	1786	61.3	66.9	9 7.1%
DV (V)	85.0	67.8	50.3	1750	60.7	65.8	93.4%

Appendix

Table 10: Comparison showing benefts of our local cluster aggregation on LLaVA-NeXT-7B across 6 benchmarks. We report numbers for different methods (M): LLaVA-NeXT-7B (Base), Vanilla (V), VisionZip (VZ), VisionZip with our proposed clustering (VZ (C)), DUET-VLM (Vanilla) (DV (V)) and our proposed method, DUET-VLM (C) represented as (C)

Method	VQA T	POPE	SQA I	MME	Avg
Total 576 Tokens (100%)
Base	58.2	85.9	69.5	1862	100.0%
Avg 270 Tokens ( ↓ 53.1% )
PDrop	57.5	84.8	69.4	1854.0	99.2%
PDrop (S)	57.6	85.2	69.2	1862	99.4%
PDrop (all)	57.8	85.2	69.2	1862.0	99.5%

Appendix

Table 11: Comparison of various token-selection schemes applied to PyramidDrop (PDrop) on top of LLaVA-1.5-7B (Base)

Target Tokens	Dominant	Contextual
192	300	7
128	170	35
64	72	30

Appendix

Table 12: Vision token configuration used in the local clustering method for LLaVA-1.5-7B to achieve different target tokens

Target Tokens	Dominant	Contextual
640	850	175
320	360	150
160	225	30

Appendix

Table 13: Vision token configuration used in the local clustering for LLaVA-NeXT-7B to achieve different target tokens

Target Tokens	Dominant	Contextual
960	1280	256
136	160	56

Appendix

Table 14: Vision token configuration used in the local clustering for Video-LLaVA-7B to achieve different target tokens

Target Tokens	Dominant	Contextual
640	870	153
320	517	91
160	207	36

Appendix

Table 15: Vision token configuration used in local clustering for Qwen2.5-VL-7B to achieve different target token budgets. The LLM-side rank-and-drop uses

\texttt{layer\_list}=[14,21]

and

\texttt{image\_token\_ratio\_list}=[0.5,0.25]

Explore similar work

Mar 10, 2026

Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity

Vision-language models (VLMs) face significant computational inefficiencies caused by excessive generation of visual tokens. While prior work shows that a large fraction of visual tokens are redundant, existing compression methods struggle to balance importance preservation and information diversity. To address this, we propose PruneSID, a training-free Synergistic Importance-Diversity approach featuring a two-stage pipeline: (1) Principal Semantic Components Analysis (PSCA) for clustering tokens into semantically coherent groups, ensuring comprehensive concept coverage, and (2) Intra-group Non-Maximum Suppression (NMS) for pruning redundant tokens while preserving key representative tokens within each group. Additionally, PruneSID incorporates an information-aware dynamic compression ratio mechanism that optimizes token compression rates based on image complexity, enabling more effective average information preservation across diverse scenes. Extensive experiments demonstrate state-of-the-art performance, achieving 96.3% accuracy on LLaVA-1.5 with only 11.1% token retention, and 92.8% accuracy at extreme compression rates (5.6%) on LLaVA-NeXT, outperforming prior methods by 2.5% with 7.8

\times

faster prefilling speed compared to the original model. Our framework generalizes across diverse VLMs and both image and video modalities, showcasing strong cross-modal versatility. Code is available at https://github.com/ZhengyaoFang/PruneSID.

Zhengyao Fang, Pengyuan Lyu, Chengquan Zhang +3

Harbin Institute of Technology, Shenzhen · Independent Researcher · Peng Cheng Laboratory

Visual Token Pruning Visual Token Compression Efficient VLM Inference Video Token Compression

Ask PDF

May 28, 2026

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning

Vision-language models (VLMs) rely on long visual token sequences for visual understanding, making the prefill stage expensive in both computation and memory. Most existing pruning methods follow an absolute-ranking paradigm, assigning importance scores to visual tokens and retaining a fixed top-K subset. In this work, we argue that this paradigm is fundamentally brittle: attention sinks distort token importance rankings, while image redundancy and query-dependent visual evidence make fixed token budgets unreliable across inputs. We propose OccamToken, a training-free framework that replaces absolute token ranking with register-anchored relative evidence testing. Instead of asking which tokens are globally important, OccamToken evaluates whether a visual token provides information beyond a register-based reference. Our key insight is that register tokens naturally absorb low-information attention patterns, making them a stable reference for identifying genuinely informative visual evidence. Based on this principle, OccamToken performs both image-adaptive redundancy pruning and query-adaptive relevance pruning through dynamic thresholds derived from register attention. Across LLaVA-NeXT, LLaVA-v1.5, and Qwen3-VL, OccamToken consistently improves the accuracy-efficiency trade-off without additional training. Notably, on LLaVA-NeXT, it reduces 2,880 visual tokens to approximately 40 while preserving over 93% of the original accuracy, enabling stable visual token compression even in the extreme 1.4% retention regime.

Geng Li, Guohao Chen, Ting Chen +6

1Nanyang Technological University (NTU)

Efficient VLM Inference Large Vision-Language Models Visual Token Pruning Visual Token Compression

Ask PDF

Jul 15, 2026

Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models

Vision-Language Models (VLMs) have achieved strong performance in multimodal understanding, yet remain challenging to deploy on resource-constrained edge devices due to the substantial computational overhead of processing numerous visual tokens. Token reduction is a promising direction for accelerating VLMs inference, but existing approaches either rely on attention maps that are incompatible with modern acceleration frameworks or depend on computationally intensive pairwise similarity comparisons, which undermine scalability and negate their practical benefits in deployment. In this paper, we propose an attention-free and lightweight token reduction framework as a plug-and-play module for VLMs, which preserves both important and diverse tokens to produce a compact visual representation. First, to enable attention-free importance estimation, we adopt an information-theoretic perspective and quantify token information using a novel entropy-based criterion, retaining those with more expressive and less degenerate feature representations. Second, to ensure diverse visual coverage in a lightweight manner, we introduce a transformation-induced consistency signal where similar tokens yield similar signals, such that sorting by this signal places similar tokens close to each other and enables stride-based selection to produce a diverse token set. Extensive experiments across multiple VLMs benchmarks demonstrate that our framework achieves a favorable accuracy-efficiency trade-off, maintaining competitive performance under aggressive compression.

Xuanyi Hao, Zuoyuan Zhang, Zhibo Wang +4

State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou 310027, China · Hong Kong University of Science and Technology, Hong Kong SAR, China · School of Artificial Intelligence, Nanchang University, Nanchang 330038, China

Vision-Language Models Efficient VLM Inference Visual Token Pruning Visual Token Compression

Ask PDF