Network pruning is an effective technique for enabling lightweight Large Vision-Language Models (LVLMs), which primarily incorporates both weights and activations into the importance metric. However, existing efforts typically process calibration data from different modalities in a unified manner, overlooking modality-specific behaviors. This raises a critical challenge: how to address the divergent behaviors of textual and visual tokens for accurate pruning of LVLMs. To this end, we systematically investigate the sensitivity of visual and textual tokens to the pruning operation by decoupling their corresponding weights, revealing that: (i) the textual pathway should be calibrated via text tokens, since it exhibits higher sensitivity than the visual pathway; (ii) the visual pathway exhibits high redundancy, permitting even 50% sparsity. Motivated by these insights, we propose a simple yet effective Asymmetric Text-Visual Weight Pruning method for LVLMs, dubbed ATV-Pruning, which establishes the importance metric for accurate weight pruning by selecting the informative tokens from both textual and visual pathways. Specifically, ATV-Pruning integrates two primary innovations: first, a calibration pool is adaptively constructed by drawing on all textual tokens and a subset of visual tokens; second, we devise a layer-adaptive selection strategy to yield important visual tokens. Finally, extensive experiments across standard multimodal benchmarks verify the superiority of our ATV-Pruning over state-of-the-art methods.
Figures & tables
Figure 1: Illustration of the divergent statistical characteristics across different modalities , manifesting as: (a) activation representation : the textual and visual activations occupy distinct clustered regions in the representation space (t-SNE visualization); and (b) pruning importance : the pruning masks derived from the text-only and visual-only calibration data exhibit a broad IoU distribution (taking 50% sparsity level as an example)
Figure 2: Modality decoupling via MoT probe. For each Transformer block, the QKV and FFN layers are replicated into visual and textual pathways , which process their respective token types. Independent pruning masks are derived for each pathway using activation statistics from text-only , image-only , or mixed calibration pools. This setup enables controlled comparison of modality-specific pruning sensitivity.
Pathway
Calibration
MMB
SQA img
VizWiz
Average
Dense
72.16
73.28
57.65
100%
50% Sparsity
Text
Text
65.03
70.55
62.94
98.26%
Visual
61.77
67.67
60.56
94.33%
Mixed
64.00
69.01
60.36
95.86%
Visual
Text
71.39
74.17
58.03
100.27%
Table 1: Sensitivity analysis of both modality pathways across different calibration token sources, benchmarks, and sparsity.
Figure 3: Overview of ATV-Pruning; Color intensity reflects the degree of visual saliency. Blocks with higher visual saliency keep more salient visual tokens, with all text tokens.
Methods
GQA
MMB
MME
MMMU
OKVQA
POPE
SQA img
TextVQA
VizWiz
Average
LLaVA-NeXT 8B, Dense
65.34
72.16
1965.12
40.11
60.13
87.84
73.28
65.42
57.65
100%
LLaVA-NeXT 8B
50% Sparsity
SparseGPT [ 12 ]
62.86
65.38
1742.80
36.00
38.17
89.23
68.12
64.21
60.12
91.74%
Wanda [ 30 ]
62.61
63.83
1620.02
35.56
25.38
88.02
69.06
64.04
60.73
88.36%
TAMP [ 17 ]
63.15
64.86
1732.54
34.78
43.59
88.04
70.00
62.78
62.50
92.67%
ATV-Pruning (Ours)
63.28
65.29
1801.51
36.11
45.34
88.22
69.86
62.76
63.34
94.00%
Table 2: Comparison results with state-of-the-art approaches on nine benchmarks under unstructured, uniform pruning. The official metric for each benchmark and average performance retention (denoted as Average) across all benchmarks, are reported to evaluate the performance of pruned models. Best results are highlighted in green
Figure 4: Ablation study about global scaling factor α . Larger α allocates more salient visual tokens. A moderate range α∈[0.5,2] yields stable performance across benchmarks.
Signal
Adaptive
MMB
SQA img
VizWiz
Average
Dense
72.16
73.28
57.65
100%
Random
✗
64.52
69.26
62.59
97.50%
Drift
✗
64.95
69.56
63.30
98.24%
Our ATV-Pruning
Drift
✓
65.29
69.86
63.34
98.56%
ABS
✓
65.12
70.65
62.59
98.41%
Table 3: Discussion on selection strategy of visual tokens. All methods select the same total number of visual tokens for calibration. “Adaptive” indicates whether the visual-token budget is dynamically distributed across blocks. Our default configuration achieves the best overall retention.
Figure 5: Effect of textual token selection under varied retained ratios during calibration , with the default visual token subset of ATV-Pruning. Removing text tokens consistently harms the performance, highlighting their necessity for stable pruning.
Method
Time (s) ↓
Rel. time ↓
Gain (pp) ↑
Wanda [ 30 ]
73.7
1.00×
+0
SparseGPT [ 12 ]
666.0
9.04 ×
+3.38
TAMP [ 17 ]
1418.4
19.25 ×
+4.31
ATV-Pruning (Ours)
99.6
1.35 ×
+5.64
Table 4: Pruning efficiency analysis , on LLaVA-NeXT (8B) at 50% sparsity. Time (s): wall-clock pruning time in seconds; Rel. time: pruning time relative to Wanda; Gain (pp): average performance gain in percentage points (pp), taken from Tab. 2 .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Methods
GQA
MMB
MME
MMMU
OKVQA
POPE
SQA img
TextVQA
VizWiz
Average
LLaVA-NeXT 8B, Dense
65.34
72.16
1965.12
40.11
60.13
87.84
73.28
65.42
57.65
100%
LLaVA-NeXT 8B
2:4 Semi-Structured Sparsity
SparseGPT [ 12 ]
55.13
42.61
1249.34
28.89
16.79
88.21
34.71
58.58
53.88
70.86%
Wanda [ 30 ]
54.67
37.80
1370.97
28.00
11.07
87.52
58.08
55.84
55.01
72.62%
TAMP [ 17 ]
56.31
42.44
1401.91
28.67
13.84
87.46
57.76
56.11
57.15
74.90%
ATV-Pruning (Ours)
57.14
46.82
1466.78
29.56
17.91
87.11
52.16
55.71
58.01
76.29%
Appendix
Table A5 : Semi-structured ( 2:4; 4:8 ) pruning comparisons on LLaVA-NeXT (8B). Best results are highlighted in green .
Methods
GQA
MMB
MME
MMMU
OKVQA
POPE
SQA img
TextVQA
VizWiz
Average
LLaVA-OneVision 7B, Dense
62.25
80.84
1998.62
49.22
61.00
89.13
95.93
76.05
60.38
100%
LLaVA-OneVision 7B
60% Sparsity
SparseGPT [ 12 ]
58.56
73.71
1538.61
37.33
44.65
89.72
79.03
72.62
62.72
88.19%
Wanda [ 30 ]
54.86
72.34
1293.20
37.89
25.28
89.54
78.58
69.64
61.00
78.25%
TAMP [ 17 ]
57.64
73.20
1571.36
39.67
37.65
89.54
78.58
69.64
61.00
86.56%
ATV-Pruning (Ours)
58.05
74.57
1406.75
40.11
46.15
89.78
80.47
70.23
61.98
88.07%
Appendix
Table A6 : Additional pruning results for LLaVA-OneVision and Qwen2.5-VL at 60% unstructured sparsity, serving as an extension of “Table 2”. Best results are highlighted in green . SQA img results for Qwen2.5-VL are omitted because the current lmms-eval setup does not yield reliable evaluations.
Figure A6 : Trend of ATV-Pruning’s block-wise Visual Drift and Adaptive Token Allocation on LLaVA-NeXT (8B) with α=1.0 . The left y-axis measures the average visual drift per block. The right y-axis indicates the corresponding average number of visual tokens selected per sample. The trend illustrates the block-adaptive nature of our method: more tokens are retained in layers where visual representations undergo significant updates.
Large Vision Language Models show impressive performance across image and video understanding tasks, yet their computational cost grows rapidly with the number of visual tokens. Existing token pruning methods mitigate this issue through empirical approaches while overlooking the internal mechanism of attention. In this paper, we propose a novel training free token pruning framework grounded in the dual form perspective of attention. We reformulate attention as an implicit linear layer whose weight matrix is the sum of rank 1 outer products, each generated by a single token's key value pair. Token pruning thus reduces to selecting an optimal subset of these rank 1 updates that best approximates the original dual weight matrix. Extending this perspective to standard softmax attention in LVLMs, we derive a novel metric quantifying both a token's information magnitude and information duplication. To efficiently select the subset with the proposed metric, we introduce Progressive Chunked Maximal Marginal Relevance. Extensive experiments demonstrate that our method achieves a better trade off between performance and efficiency, while providing another perspective on existing pruning approaches.
While Vision-Language Models (VLMs) have demonstrated remarkable performance in processing and understanding both text and images, their large parameter sizes lead to significant computational overhead, limiting their deployment on resource-constrained devices. While pruning has been effective for compressing Large Language Models (LLMs), directly applying it to VLMs leads to significant performance drops, largely due to redundant visual tokens interfering with importance estimation. To this end, we propose SlimVLM, a structured pruning framework designed to compress VLMs while preserving their task performance. We introduce an adaptive visual token selection strategy for VLMs that leverages average text-to-visual attention scores to assess the importance of visual tokens, removing redundant ones during pruning based on a set threshold, thereby optimizing the importance calculation. Recognizing the varying tolerance to sparsity across different modules, we also propose a Sensitivity-aware dynamic pruning mechanism that determines the appropriate pruning ratio for each module by calculating the linear reconstruction error between the outputs of the pruned and unpruned modules, ensuring overall performance stability. Experimental results show that SlimVLM outperforms existing methods across multiple multimodal benchmarks, achieving state-of-the-art performance.
Large Vision-Language Models (LVLMs) rely on dense visual tokens to capture fine-grained visual information, but processing all these tokens incurs substantial computational and memory overhead during inference. To address this issue, we propose ResPrune, a training-free visual token pruning framework that enables efficient LVLM inference by selecting a compact yet informative subset of visual tokens. ResPrune formulates visual token pruning as a subspace reconstruction problem and employs a greedy subspace expansion strategy guided by residual energy, allowing it to preserve the geometric structure of the original visual token space. To further incorporate cross modal alignment, the selection process is conditioned on textual relevance, encouraging the retention of tokens that are both informative and instruction-relevant. The proposed method is lightweight and model-agnostic, and can be seamlessly integrated into existing LVLM pipelines without retraining or architectural modifications. Extensive experiments on multiple LVLM backbones, including LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL, demonstrate that ResPrune consistently outperforms existing pruning approaches across a wide range of benchmarks, while achieving effective reductions in computation, memory consumption, and inference latency.
Xu Li, Yi Zheng, Yuxuan Liang +5
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai 200433, China