CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

Yunkai Dang; Yizhu Jiang; Yifan Jiang; Qi Fan; Yinghuan Shi; Wenbin Li; Yang Gao

cs.CVApr 14, 2026

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

Authors: Yunkai Dang, Yizhu Jiang, Yifan Jiang, Qi Fan, Yinghuan Shi, Wenbin Li, Yang Gao

Organizations: School of Artificial Intelligence Science and Technology, Nanjing University.

Abstract

Multimodal Large Language Models (MLLMs) suffer from substantial computational overhead due to the high redundancy in visual token sequences. Existing approaches typically address this issue using single-layer Vision Transformer (ViT) features and static pruning strategies. However, such fixed configurations are often brittle under diverse instructions. To overcome these limitations, we propose CLASP, a plug-and-play token reduction framework based on class-adaptive layer fusion and dual-stage pruning. Specifically, CLASP first constructs category-specific visual representations through multi-layer vision feature fusion. It then performs dual-stage pruning, allocating the token budget between attention-salient pivot tokens for relevance and redundancy-aware completion tokens for coverage. Through class-adaptive pruning, CLASP enables prompt-conditioned feature fusion and budget allocation, allowing aggressive yet robust visual token reduction. Extensive experiments demonstrate that CLASP consistently outperforms existing methods across a wide range of benchmarks, pruning ratios, and MLLM architectures. Code will be available at https://github.com/Yunkaidang/CLASP.

Figures & tables

Figure 1: Impact of hyperparameter settings on MMVet dataset performance (LLaVA-v1.5-7B, 192 retained tokens). The heatmaps illustrate the score distribution across different question categories under varying conditions. Top: Evaluation of five representative layer-fusion strategies (A–E), ordered by an increasing proportion of weights assigned to deeper layers ( i.e. , shifting from shallow in A to deep in E). Bottom: Sweeping the attention–similarity mixing coefficient (higher: more attention; lower: more similarity).

Figure 2: Overview of our method. The framework utilizes a prompt-to-class router to condition visual processing on textual intent. (i) Class-Adaptive Layer Fusion: ViT features are aggregated from multiple layers using class-specific mixture weights to capture an appropriate level of visual abstraction. (ii) Class-Adaptive Pruning: The projected visual token budget $R$ is dynamically split (ratio $a$ ) between attention-based selection (Stage 1) and similarity-based clustering (Stage 2) to balance instruction relevance with visual coverage.

Methods	GQA	MMB	MME	POPE	SQA	VQA ${}_{\text{V2}}$	VQA ${}_{\text{Text}}$	Average
Upper Bound, 576 Tokens	61.9	64.7	1862	85.9	69.5	78.4	58.2	100.0%
LLaVA-1.5 7B	Retain 192 Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 66.7\%)}}$
ToMe (ICLR23) ( Bolya et al., 2022 )	54.3	60.5	1563	72.4	65.2	68.0	52.1	88.5%
FastV (ECCV24) ( Chen et al., 2024a )	52.7	61.2	1612	64.8	67.3	67.1	52.5	87.8%
MustDrop (2024.11) ( Liu et al., 2024c )	58.2	62.3	1787	82.6	69.2	76.0	56.5	96.6%
LLaVA-PruMerge (ICCV25) ( Shang et al., 2025 )	54.3	59.6	1632	71.3	67.9	70.6	54.3	90.2%

Table 1: Performance comparison of various methods on LLaVA-v1.5-7B across different benchmarks. Results are shown for different pruning ratios, with accuracy and average performance highlighted. Best results in blue .

Methods	GQA	MMB	MME	POPE	SQA	VQA ${}_{\text{V2}}$	VQA ${}_{\text{Text}}$	Average
Upper Bound, 2880 Tokens	64.2	67.4	1851	86.5	70.1	81.8	64.9	100.0%
LLaVA-NeXT 7B	Retain 320 Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 88.9\%)}}$
FastV (ECCV24) ( Chen et al., 2024a )	55.9	61.6	1661	71.7	62.8	71.9	55.7	88.0%
LLaVA-PruMerge (ICCV25) ( Shang et al., 2025 )	53.6	61.3	1534	60.8	66.4	69.7	50.6	85.6%
PDrop (CVPR25) ( Xing et al., 2024 )	56.4	63.4	1663	77.6	67.5	73.5	54.4	90.9%
MustDrop (2024.11) ( Liu et al., 2024c )	57.3	62.8	1641	82.1	68.0	73.7	59.9	92.2%

Table 2: Performance comparison of various methods on LLaVA-NeXT-7B across different benchmarks. Results are shown for different pruning ratios, with accuracy and average performance highlighted. Best results in blue .

Methods	MME	POPE	SQA	VQA ${}_{\text{Text}}$	MMB	Avg
Upper Bound	2308	86.1	78.0	77.8	82.2	100%
Qwen2.5-VL-7B	TokenPruningRate = 66.7%
SparseVLM	2165	81.7	73.8	71.5	78.4	94.1%
CLASP (ours)	2197	85.1	75.4	74.7	78.6	96.5%
Qwen2.5-VL-7B	TokenPruningRate = 77.8%
SparseVLM	2086	77.7	72.8	68.7	75.6	90.8%

Table 3: Performance comparison on Qwen2.5-VL-7B across widely-used benchmarks. Best results in blue .

Method	TGIF	MSVD	MSRVTT	Average
Upper Bound	43.47	61.93	51.28	52.23
SparseVLM	44.67	59.29	49.69	51.22
CLASP (ours)	45.65	61.62	51.58	52.95

Table 4: Performance comparison on Video-LLaVA-7B across video benchmarks (TGIF, MSVD, MSRVTT). Best results in blue .

Method	Time	Prefill	Latency	Mem.	Acc.	$\Delta$
Retain 192 Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 66.7\%)}}$
Upper Bound	49:41	0.5ms	0.334s	19.0G	100.0%	-
SparseVLM	40:51	0.6ms	0.251s	15.8G	97.3%	1.2 $\times$
CLASP (ours)	33:08	0.2ms	0.216s	17.6G	99.6%	1.5 $\times$
Retain 58 Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 89.9\%)}}$
Upper Bound	49:41	0.5ms	0.334s	19.0G	100.0%	-

Table 5: Efficiency analysis on POPE at different retention levels.

\Delta

denotes the speedup ratio in Time.

Figure 3: Example visualization of the original image and the corresponding token-retention map.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Category	Fusion weights (LLaVA family)	Fusion weights (Qwen2.5-VL)	Split ratio $a_{c}$
0	Object identification	$\{L_{5}\!:\!0.2,\;L_{15}\!:\!0.3,\;L_{22}\!:\!0.5\}$	$\{L_{9}\!:\!0.2,\;L_{22}\!:\!0.3,\;L_{31}\!:\!0.5\}$	0.8
1	Attribute / breed identification	$\{L_{5}\!:\!0.2,\;L_{22}\!:\!0.8\}$	$\{L_{9}\!:\!0.2,\;L_{31}\!:\!0.8\}$	0.4
2	Text / symbol recognition	$\{L_{5}\!:\!0.2,\;L_{22}\!:\!0.8\}$	$\{L_{9}\!:\!0.2,\;L_{31}\!:\!0.8\}$	0.7
3	Scene understanding	$\{L_{20}\!:\!0.2,\;L_{22}\!:\!0.8\}$	$\{L_{28}\!:\!0.2,\;L_{31}\!:\!0.8\}$	0.7
4	Spatial relations	$\{L_{14}\!:\!0.2,\;L_{17}\!:\!0.3,\;L_{22}\!:\!0.5\}$	$\{L_{21}\!:\!0.2,\;L_{24}\!:\!0.3,\;L_{31}\!:\!0.5\}$	0.7
5	Counting	$\{L_{5}\!:\!0.2,\;L_{15}\!:\!0.3,\;L_{22}\!:\!0.5\}$	$\{L_{9}\!:\!0.2,\;L_{22}\!:\!0.3,\;L_{31}\!:\!0.5\}$	0.6

Appendix

Table 6: Hyperparameters for class-adaptive layer fusion and pruning across MLLM backbones. Fusion weights are reported as {layer: coefficient}. The split ratio

a_{c}

governs the prompt-conditioned budget allocation between attention-salient pivots for relevance and redundancy-aware completion tokens for coverage.

ID	Category	GQA	MMB	MME	MMVet	POPE	SQA	VQA ${}_{\text{v2}}$	VQA ${}_{\text{Text}}$
0	Object identification	2095	272	355	21	2277	178	18263	239
1	Attribute / breed identification	4015	268	85	19	0	410	19380	1037
2	Text / symbol recognition	75	111	147	12	0	465	4368	2856
3	Scene understanding	2319	1950	1209	58	6633	521	35553	388
4	Spatial relations	3632	379	70	18	0	193	7378	39
5	Counting	9	100	87	5	0	54	10007	53

Appendix

Table 7: Detailed statistical distribution of sample counts per task category across all evaluated multimodal benchmarks.

Calibration Size	Sampling Strategy	MME	TextVQA	MMVet
4.5k	Balanced (Avg)	$1814\pm 6$	$56.6\pm 0.3$	$32.0\pm 0.3$
4.5k	Random	$1809\pm 15$	$56.4\pm 0.6$	$32.1\pm 0.4$
9k	Balanced (Avg)	$1845\pm 3$	$57.5\pm 0.1$	$33.2\pm 0.1$
9k	Random	$1840\pm 5$	$57.6\pm 0.2$	$33.2\pm 0.2$
18k	Balanced (Avg)	$1845\pm 2$	$57.5\pm 0.1$	$33.3\pm 0.0$
18k	Random	$1845\pm 3$	$57.6\pm 0.1$	$33.2\pm 0.1$

Appendix

Table 8: Ablation on calibration set size and sampling strategy. Results are reported as mean

\pm

standard deviation across 5 random seeds. The framework achieves optimal, low-variance performance with a balanced dataset of merely 9k samples, demonstrating excellent sample efficiency.

Component	Time Cost	Memory Usage
Intent Classifier (Online Inference)	$<2$ ms	15.5 GB
Calibration of Layer Weights ( $W$ ) (Offline)	$\approx 1.5$ h	19.4 GB
Calibration of Split Ratios ( $a$ ) (Offline)	$\approx 1.0$ h	17.6 GB

Appendix

Table 9: Time and memory cost of offline calibration and online routing. The offline calibration of layer weights and split ratios takes only a few hours on a single GPU without requiring backpropagation. The online intent classifier introduces negligible latency.

Figure 4: Per-benchmark performance under increasing token pruning. We plot performance as a function of pruning ratio on eight evaluation suites, comparing our method with representative pruning baselines (SparseVLM and PDrop). The dashed line denotes the unpruned model performance. Our method exhibits the slowest degradation and stays consistently closest to the unpruned upper bound, particularly at aggressive pruning ratios.

Config	MME	TextVQA	GQA	POPE
Baseline	1721	56.1	57.6	83.6
+ Fusion	1814	56.9	59.6	85.0
+ Ratio	1787	56.6	58.4	84.7
+ Both (CLASP)	1848	57.6	60.4	85.6

Appendix

Table 10: Ablation study of core components. We evaluate the individual and synergistic effects of Class-Adaptive Layer Fusion (+Fusion) and Category-Dependent Split Ratio (+Ratio) on LLaVA-v1.5-7B (

R=192

). Both components contribute significantly, and their combination yields the best overall performance.

Methods	GQA	MMB	MME	POPE	SQA	VQA ${}_{\text{Text}}$	Average
Upper Bound	63.3	68.9	1818	85.9	72.8	61.8	100.0%
LLaVA-1.5 13B	Retain 192 Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 66.7\%)}}$
SparseVLM	58.7	67.4	1768	82.2	73.1	59.6	96.9%
CLASP (ours)	60.1	68.0	1785	86.2	72.8	59.6	98.2%
LLaVA-1.5 13B	Retain 128 Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 77.8\%)}}$
SparseVLM	57.9	65.8	1774	81.1	69.9	58.4	95.0%

Appendix

Table 11: Performance comparison of various methods on LLaVA-v1.5-13B across different benchmarks. Results are shown for different pruning ratios, with accuracy and average performance highlighted. Best results in blue .

Methods	GQA	MMB	MME	POPE	SQA	VQA ${}_{\text{Text}}$	Average
Upper Bound	64.2	67.4	1851	86.5	70.1	64.9	100.0%
LLaVA-NeXT 7B	Retain 640 Tokens
SparseVLM	61.2	62.2	1697	85.3	67.6	59.7	94.4%
CLASP (ours)	63.1	62.8	1746	87.9	69.8	61.8	97.0%
LLaVA-NeXT 7B	Retain 320 Tokens
SparseVLM	56.1	60.6	1533	82.4	66.1	58.4	90.0%

Appendix

Table 12: Performance comparison of various methods on LLaVA-NeXT-7B across different benchmarks. Results are shown for different pruning ratios, with accuracy and average performance highlighted. Best results in blue .

Methods	GQA	MMB	MME	POPE	SQA	VQA ${}_{\text{Text}}$	Average
Upper Bound	65.4	70.0	1901	86.2	73.5	64.3	100.0%
LLaVA-NeXT 13B	Retain 640 Tokens
SparseVLM	62.7	62.0	1821	86.0	71.5	59.3	94.9%
CLASP (ours)	64.2	63.2	1867	86.8	73.5	61.5	97.2%
LLaVA-NeXT 13B	Retain 320 Tokens
SparseVLM	60.9	60.8	1798	83.5	70.6	57.2	92.7%

Appendix

Table 13: Performance comparison of various methods on LLaVA-NeXT-13B across different benchmarks. Results are shown for different pruning ratios, with accuracy and average performance highlighted. Best results in blue .

Methods	TextVQA	MME	GQA	MMVet
Upper Bound	82.5	2270	64.9	64.0
InternVL2 26B	Retain 35% Tokens
FastV	75.6	2140	61.2	45.0
ToMe	75.7	2178	63.6	52.5
CLASP (ours)	81.7	2262	65.0	63.2

Appendix

Table 14: Performance comparison on InternVL2-26B across widely-used benchmarks. Results are shown for an aggressive pruning ratio (retaining

35\%

of visual tokens), with accuracy highlighted. Best results in blue .

Layers	Class 0	Class 1	Class 2	Class 3	Class 4	Class 5	Class 6	Class 7	Class 8	Avg
	MMVet ( $R=192$ )
A	28.0	28.2	28.6	28.5	27.8	28.4	28.3	28.7	28.3	28.3
B	28.9	28.3	27.1	28.8	16.2	27.8	28.2	28.1	28.2	26.8
C	28.4	28.6	25.8	28.5	14.0	28.1	28.9	28.4	28.4	26.6
D	28.7	28.0	29.1	28.3	28.3	29.5	28.8	28.5	28.2	28.6
E	28.2	28.3	27.9	28.4	28.6	27.9	28.1	28.0	28.2	28.2

Appendix

Table 15: Ablation on layer-mixture strategies across different task categories with

R=192

. The columns correspond to specific task types: (0) Object Identification, (1) Attribute/Breed ID, (2) Text/Symbol Recognition, (3) Scene Understanding, (4) Spatial Relations, (5) Counting, (6) Action/Interaction, (7) Intention/Function, and (8) Default. Layer Mixture Legend: A:

0.2L_{2}+0.3L_{6}+0.5L_{11}

; B:

0.2L_{5}+0.3L_{15}+0.5L_{22}

; C:

0.2L_{12}+0.3L_{15}+0.5L_{19}

; D:

0.2L_{5}+0.8L_{22}

; E:

0.2L_{20}+0.8L_{22}

. Best results are highlighted in blue , and the row with the highest average score in each dataset is shaded.

Budget	Method	LLaVA-B	MMVet	VizWiz	MMB-CN	MMB	SEED
Unpruned	LLaVA-v1.5-7B	66.8	30.9	50.0	58.1	64.7	66.2
R=192	SparseVLM	66.1	33.1	50.5	53.7	62.5	64.2
R=192	CLASP (Ours)	66.7	33.3	52.1	57.9	61.3	65.4
R=128	SparseVLM	62.7	29.0	51.4	51.1	60.0	63.6
R=128	CLASP (Ours)	65.6	30.0	51.9	57.1	60.7	63.0
R=64	SparseVLM	57.5	24.9	50.1	46.1	56.2	56.8

Appendix

Table 16: Supplementary evaluations on diverse open datasets using LLaVA-v1.5-7B under varying token budgets. We compare our CLASP framework against SparseVLM. The best results between the two pruning methods at each budget are highlighted in bold.

$p$	Class 0	Class 1	Class 2	Class 3	Class 4	Class 5	Class 6	Class 7	Class 8
	MME ( $R=192$ )
0.1	1815	1810	1815	1812	1815	1782	1812	1812	1813
0.3	1809	1814	1770	1814	1818	1802	1805	1809	1804
0.5	1799	1813	1791	1812	1811	1801	1813	1815	1811
0.7	1817	1812	1813	1828	1818	1812	1812	1814	1813
0.9	1816	1813	1809	1816	1817	1816	1813	1814	1814

Appendix

Table 17: Ablation study on the sensitivity of different task categories to the attention selection ratio

p

(with fixed

R=192

). The columns correspond to specific task types mapped as follows: (0) Object Identification, (1) Attribute/Breed ID, (2) Text/Symbol Recognition, (3) Scene Understanding, (4) Spatial Relations, (5) Counting, (6) Action/Interaction, (7) Intention/Function, and (8) Default. Best results are highlighted in blue , and the row with the robust average score in each dataset is shaded.

Figure 5: Layer mixture ablation under a fixed token budget ( $R=192$ ) on MME, TextVQA and SQA. Rows are layer mixture strategies with weights in parentheses. Columns are question types: C0 object identification, C1 attribute or breed identification, C2 text or symbol recognition, C3 scene understanding, C4 spatial relations, C5 counting, C6 action or interaction, C7 intention or function, C8 default.

Figure 6: Ablation on attention and similarity mixture weight under a fixed token budget ( $R=192$ ) on MME, TextVQA and SQA. Rows are the mixture ratios. Columns are question types: C0 object identification, C1 attribute or breed identification, C2 text or symbol recognition, C3 scene understanding, C4 spatial relations, C5 counting, C6 action or interaction, C7 intention or function, C8 default.

Iterations	MME	GQA	MMVet	POPE	SQA	TextVQA
3	1828	60.41	31.1	85.57	69.58	56.83
5	1848	60.44	31.8	85.55	69.56	57.59
7	1823	59.08	31.3	85.48	68.32	56.81
9	1762	59.02	30.5	84.33	68.37	56.92
11	1757	58.61	30.1	84.41	68.22	56.94

Appendix

Table 18: Impact of the number of clustering iterations on performance (

R=192

Methods	MMBench	MMVet	MME	TextVQA	VQA-v2	GQA	SQA	POPE
Search-derived Weights (fixed $W$ )	61.34	33.3	1848	57.59	77.08	60.44	69.56	85.55
Search-derived Weights init + Calibrated $W$	60.13	32.2	1759	56.15	77.08	60.22	69.39	85.58
Penultimate-biased init + Calibrated $W$	57.7	33.2	1722	56.05	77.14	59.78	68.69	85.67
Uniform init + Calibrated $W$	17.70	12.8	1127	30.28	62.43	43.13	47.63	78.03

Appendix

Table 19: Effect of initialization under a fixed budget (

R{=}192

). We compare our search-derived weight configuration with three initialization schemes while keeping the training protocol and budget fixed. Evaluation covers three general multimodal benchmarks (MMBench, MMVet, MME), four VQA/QA benchmarks (TextVQA, VQA-v2, GQA, SQA), and the hallucination benchmark POPE. Higher is better for all metrics. Best and second-best results are bold and underlined , respectively.

Budget	Method	MME	GQA	POPE	TextVQA
Unpruned	LLaVA-v1.5-7B	1862	61.9	85.9	58.2
R=192	Static [3, 18, 24]	1756	57.4	83.9	57.6
R=192	CLASP (Ours)	1848	60.4	85.6	57.6
R=128	Static [3, 18, 24]	1701	56.2	82.7	55.3
R=128	CLASP (Ours)	1790	58.9	85.2	56.7
R=64	Static [3, 18, 24]	1614	54.1	78.2	51.9

Appendix

Table 20: Performance comparison between the optimal Static Fusion baseline and our Class-Adaptive Fusion (CLASP) on LLaVA-v1.5-7B under varying token budgets. Best results are highlighted in bold.

Retention Budget ( $R$ )	TGIF	MSVD	MSRVTT
$50\%$	45.53 $\pm$ 0.18	61.72 $\pm$ 0.21	51.45 $\pm$ 0.13

Appendix

Table 21: Variance analysis of CLASP on video benchmarks (TGIF, MSVD, MSRVTT). Results are reported as mean

\pm

standard deviation across five random seeds (

42,43,44,45,46

) under a

50\%

retention budget.

Forced “Default” Ratio	MME	TextVQA	SQA	MMVet	POPE
$5\%$	1840	57.4	69.5	33.3	85.5
$20\%$	1799	55.0	68.7	32.8	84.7
$50\%$	1756	55.1	68.0	31.3	83.6
$100\%$	1715	53.7	66.7	29.6	82.2
Random Error Ratio	MME	TextVQA	SQA	MMVet	POPE
$5\%$	1834	57.4	69.4	33.1	85.3

Appendix

Table 22: Ablation on router robustness and classification errors. We compare the impact of forcing a specific ratio of samples to the “Default” class (top) versus randomly misclassifying them (bottom). Results demonstrate that falling back to the default class under uncertainty is significantly safer than random assignment.

Retained Tokens ( $R$ )	Total Time	Memory	FLOPs	Total Prefill	Classifier Time	Weight Fusion	K-means
192	223.1 ms	17.62 GB	2.67 T	125.2 ms	$<2$ ms	$<0.5$ ms	27.2 ms
128	201.7 ms	17.59 GB	1.97 T	109.6 ms	$<2$ ms	$<0.5$ ms	23.1 ms
64	155.8 ms	17.45 GB	1.39 T	74.7 ms	$<2$ ms	$<0.5$ ms	17.2 ms

Appendix

Table 23: Detailed latency and computational overhead breakdown. We report the end-to-end Total Time, Memory footprint, FLOPs, and the specific time consumed by distinct pipeline components (Total Prefilling, Classifier, Weight Fusion, and K-means) under varying retention budgets (

R

). The overhead of our proposed components is minimal compared to the overall prefilling savings.

Router Model	Count Variance	MME	TextVQA	SQA	MMVet	POPE
Qwen3-8B (Default)	0	1848	57.6	69.6	33.3	85.6
Llama3-8B	37.2	1840	57.5	69.4	32.9	85.7
Qwen3-32B	7.9	1848	57.7	69.4	33.1	85.3

Appendix

Table 24: Performance comparison of CLASP across different prompt-to-class router models. We report the Count Variance (measuring the intent assignment deviation relative to the default Qwen3-8B router) alongside task accuracy across five representative benchmarks. The downstream performance remains highly stable regardless of the chosen router.

Method	MME	TextVQA	SQA	MMVet	POPE
Nuwa (ICLR 2026) ( Huang et al., 2026 )	1834	57.4	68.2	30.5	86.4
Holov (NeurIPS 2025) ( Zou et al., 2025a )	1820	57.4	69.8	-	85.6
BTP (NeurIPS 2025) ( Li et al., 2025 )	1816	-	69.1	29.1	85.6
CLASP (ours)	1848	57.6	69.6	33.3	85.6

Appendix

Table 25: Comparison with concurrent state-of-the-art methods. Evaluation is performed on LLaVA-v1.5-7B with a token retention budget of

R=192

. “-” indicates that the result is not reported by the respective authors.

Panel of Figure 7 — Figure 7: Qualitative comparisons on Cases 1–4. Each case visualizes the original image and the corresponding token-retention map results under three pruning ratios ( $R=66.7\%,77.8\%,88.9\%$ ) at layers 2, 6, and 15.

Panel of Figure 8 — Figure 8: Qualitative comparisons on Cases 5–8. Each case visualizes the original image and the corresponding token-retention map results under three pruning ratios ( $R=66.7\%,77.8\%,88.9\%$ ) at layers 2, 6, and 15.

Explore similar work

Jun 23, 2026

Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models

Reducing visual token redundancy is critical for accelerating Multimodal Large Language Models (MLLMs) without degrading cross-modal reasoning performance. Existing token pruning methods typically rely on single-layer signals, such as attention scores or token similarities, which overlook the cross-layer transformation of visual representations and may exhibit positional bias in multimodal token sequences. To address this limitation, we propose a training-free token pruning framework based on Cross-Layer Spectral Evolution (CLSE). Instead of measuring token importance from single-layer feature magnitudes, CLSE quantifies how token representations evolve across Transformer layers in the frequency domain. This evolution reflects the transition from high-frequency structural details to low-frequency semantic abstractions. We observe that tokens with stronger spectral redistribution across layers are more likely to be semantically active and should therefore be preserved. By modeling cross-layer token dynamics, CLSE provides a stable importance criterion that mitigates positional bias. Extensive experiments on both image and video benchmarks demonstrate that CLSE achieves a superior trade-off between efficiency and accuracy under aggressive token reduction. Across multiple MLLMs, CLSE reduces FLOPs, KV cache memory, and latency while maintaining competitive or improved performance.

Bin Chen, Yuxiang Cai, Yadan Luo +3

School of Software Technology, Zhejiang University, Ningbo, China · Zhejiang Key Laboratory of Digital-Intelligence Service Technology, China · The University of Queensland, St Lucia, QLD, Australia +2

Efficient Multimodal Inference Multimodal Large Language Models Token Pruning Multimodal Token Compression Visual Token Pruning

Ask PDF

Feb 27, 2026

HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit

The quadratic computational cost of processing vision tokens in Multimodal Large Language Models (MLLMs) hinders their widespread adoption. While progressive vision token pruning offers a promising solution, current methods misinterpret shallow layer functions and use rigid schedules, which fail to unlock the full efficiency potential. To address these issues, we propose HiDrop, a framework that aligns token pruning with the true hierarchical function of MLLM layers. HiDrop features two key innovations: (1) Late Injection, which bypasses passive shallow layers to introduce visual tokens exactly where active fusion begins; and (2) Concave Pyramid Pruning with an Early Exit mechanism to dynamically adjust pruning rates across middle and deep layers. This process is optimized via an inter-layer similarity measure and a differentiable top-k operator. To ensure practical efficiency, HiDrop further incorporates persistent positional encoding, FlashAttention-compatible token selection, and parallel decoupling of vision computation to eliminate hidden overhead associated with dynamic token reduction. Extensive experiments show that HiDrop compresses about 90% visual tokens while matching the original performance and accelerating training by 1.72 times. Our work not only sets a new state-of-the-art for efficient MLLM training and inference but also provides valuable insights into the hierarchical nature of multimodal fusion. The code is released at https://github.com/EIT-NLP/HiDrop.

Hao Wu, Yingqi Fan, Jinyang Dai +3

Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · University of Science and Technology of China +2

Efficient VLM Inference Large Vision-Language Models Multimodal Token Compression Visual Token Pruning

Ask PDF

Jun 30, 2026

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

Multimodal Large Language Models (MLLMs) incur prohibitive inference costs due to long visual token sequences. Training-free visual token reduction provides an efficient solution. However, existing methods distort attention distributions, giving rise to a phenomenon we term Attention Logit Collapse. To address this issue, we propose ERA, an Entropy-guided visual token pruning framework with Rectified Attention for efficient MLLMs. Specifically, ERA comprises three crucial components: Dual-view Entropy Pruning (DEP), Bias-aware Token Recycling (BTR), and Logit-preserving Attention Rectification (LAR). First, DEP identifies representative anchor tokens by jointly modeling visual diversity and head-wise saliency. BTR then recycles pruned tokens into their corresponding anchors while estimating a cluster-level logit bias. Building upon this, LAR injects the estimated bias into attention logits, effectively rectifying the collapse induced by token reduction. Together, these components preserve visual evidence even under aggressive compression, enabling robust performance across single-image, multi-image, and video settings on a wide range of MLLMs. Beyond delivering practical acceleration, ERA establishes logit-preserving visual token pruning as a principled framework for efficient MLLMs, unifying theoretical foundation, algorithmic design, and practical deployment. The code is at https://github.com/924973292/ERA.

Yuhao Wang, Mu Qiao, Haiwen Diao +5

School of Future Technology, Dalian University of Technology, Dalian 116024, China · S-Lab, Nanyang Technological University, Singapore 639798 · OPPO Research Institute, Shenzhen 518000, China +1

Efficient VLM Inference Multimodal Large Language Models Multimodal Token Compression Visual Token Pruning

Ask PDF