Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models

Sijie Li; Biao Qian; Jungong Han

cs.CVMar 16, 2026

Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models

Authors: Sijie Li, Biao Qian, Jungong Han

Organizations: University of Sheffield, UK · Tsinghua University, China

Abstract

Network pruning is an effective technique for enabling lightweight Large Vision-Language Models (LVLMs), which primarily incorporates both weights and activations into the importance metric. However, existing efforts typically process calibration data from different modalities in a unified manner, overlooking modality-specific behaviors. This raises a critical challenge: how to address the divergent behaviors of textual and visual tokens for accurate pruning of LVLMs. To this end, we systematically investigate the sensitivity of visual and textual tokens to the pruning operation by decoupling their corresponding weights, revealing that: (i) the textual pathway should be calibrated via text tokens, since it exhibits higher sensitivity than the visual pathway; (ii) the visual pathway exhibits high redundancy, permitting even 50% sparsity. Motivated by these insights, we propose a simple yet effective Asymmetric Text-Visual Weight Pruning method for LVLMs, dubbed ATV-Pruning, which establishes the importance metric for accurate weight pruning by selecting the informative tokens from both textual and visual pathways. Specifically, ATV-Pruning integrates two primary innovations: first, a calibration pool is adaptively constructed by drawing on all textual tokens and a subset of visual tokens; second, we devise a layer-adaptive selection strategy to yield important visual tokens. Finally, extensive experiments across standard multimodal benchmarks verify the superiority of our ATV-Pruning over state-of-the-art methods.

Figures & tables

Figure 1: Illustration of the divergent statistical characteristics across different modalities , manifesting as: (a) activation representation : the textual and visual activations occupy distinct clustered regions in the representation space (t-SNE visualization); and (b) pruning importance : the pruning masks derived from the text-only and visual-only calibration data exhibit a broad IoU distribution (taking 50% sparsity level as an example)

Figure 2: Modality decoupling via MoT probe. For each Transformer block, the QKV and FFN layers are replicated into visual and textual pathways , which process their respective token types. Independent pruning masks are derived for each pathway using activation statistics from text-only , image-only , or mixed calibration pools. This setup enables controlled comparison of modality-specific pruning sensitivity.

Pathway	Calibration	MMB	SQA ${}_{\text{img}}$	VizWiz	Average
Dense		72.16	73.28	57.65	100%
50% Sparsity
Text	Text	65.03	70.55	62.94	98.26%
	Visual	61.77	67.67	60.56	94.33%
	Mixed	64.00	69.01	60.36	95.86%
Visual	Text	71.39	74.17	58.03	100.27%

Table 1: Sensitivity analysis of both modality pathways across different calibration token sources, benchmarks, and sparsity.

Figure 3: Overview of ATV-Pruning; Color intensity reflects the degree of visual saliency. Blocks with higher visual saliency keep more salient visual tokens, with all text tokens.

Methods	GQA	MMB	MME	MMMU	OKVQA	POPE	SQA ${}_{\text{img}}$	TextVQA	VizWiz	Average
LLaVA-NeXT 8B, Dense	65.34	72.16	1965.12	40.11	60.13	87.84	73.28	65.42	57.65	100%
LLaVA-NeXT 8B	50% Sparsity
SparseGPT [ 12 ]	62.86	65.38	1742.80	36.00	38.17	89.23	68.12	64.21	60.12	91.74%
Wanda [ 30 ]	62.61	63.83	1620.02	35.56	25.38	88.02	69.06	64.04	60.73	88.36%
TAMP [ 17 ]	63.15	64.86	1732.54	34.78	43.59	88.04	70.00	62.78	62.50	92.67%
ATV-Pruning (Ours)	63.28	65.29	1801.51	36.11	45.34	88.22	69.86	62.76	63.34	94.00%

Table 2: Comparison results with state-of-the-art approaches on nine benchmarks under unstructured, uniform pruning. The official metric for each benchmark and average performance retention (denoted as Average) across all benchmarks, are reported to evaluate the performance of pruned models. Best results are highlighted in green

Figure 4: Ablation study about global scaling factor

\alpha

. Larger

\alpha

allocates more salient visual tokens. A moderate range

\alpha\in[0.5,2]

yields stable performance across benchmarks.

Signal	Adaptive	MMB	SQA ${}_{\text{img}}$	VizWiz	Average
Dense		72.16	73.28	57.65	100%
Random	✗	64.52	69.26	62.59	97.50%
Drift	✗	64.95	69.56	63.30	98.24%
Our ATV-Pruning
Drift	✓	65.29	69.86	63.34	98.56%
ABS	✓	65.12	70.65	62.59	98.41%

Table 3: Discussion on selection strategy of visual tokens. All methods select the same total number of visual tokens for calibration. “Adaptive” indicates whether the visual-token budget is dynamically distributed across blocks. Our default configuration achieves the best overall retention.

Figure 5: Effect of textual token selection under varied retained ratios during calibration , with the default visual token subset of ATV-Pruning. Removing text tokens consistently harms the performance, highlighting their necessity for stable pruning.

Method	Time (s) $\downarrow$	Rel. time $\downarrow$	Gain (pp) $\uparrow$
Wanda [ 30 ]	73.7	$1.00\times$	+0
SparseGPT [ 12 ]	666.0	9.04 $\times$	+3.38
TAMP [ 17 ]	1418.4	19.25 $\times$	+4.31
ATV-Pruning (Ours)	99.6	1.35 $\times$	+5.64

Table 4: Pruning efficiency analysis , on LLaVA-NeXT (8B) at 50% sparsity. Time (s): wall-clock pruning time in seconds; Rel. time: pruning time relative to Wanda; Gain (pp): average performance gain in percentage points (pp), taken from Tab. 2 .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	GQA	MMB	MME	MMMU	OKVQA	POPE	SQA ${}_{\text{img}}$	TextVQA	VizWiz	Average
LLaVA-NeXT 8B, Dense	65.34	72.16	1965.12	40.11	60.13	87.84	73.28	65.42	57.65	100%
LLaVA-NeXT 8B	2:4 Semi-Structured Sparsity
SparseGPT [ 12 ]	55.13	42.61	1249.34	28.89	16.79	88.21	34.71	58.58	53.88	70.86%
Wanda [ 30 ]	54.67	37.80	1370.97	28.00	11.07	87.52	58.08	55.84	55.01	72.62%
TAMP [ 17 ]	56.31	42.44	1401.91	28.67	13.84	87.46	57.76	56.11	57.15	74.90%
ATV-Pruning (Ours)	57.14	46.82	1466.78	29.56	17.91	87.11	52.16	55.71	58.01	76.29%

Appendix

Table A5 : Semi-structured ( 2:4; 4:8 ) pruning comparisons on LLaVA-NeXT (8B). Best results are highlighted in green .

Methods	GQA	MMB	MME	MMMU	OKVQA	POPE	SQA ${}_{\text{img}}$	TextVQA	VizWiz	Average
LLaVA-OneVision 7B, Dense	62.25	80.84	1998.62	49.22	61.00	89.13	95.93	76.05	60.38	100%
LLaVA-OneVision 7B	60% Sparsity
SparseGPT [ 12 ]	58.56	73.71	1538.61	37.33	44.65	89.72	79.03	72.62	62.72	88.19%
Wanda [ 30 ]	54.86	72.34	1293.20	37.89	25.28	89.54	78.58	69.64	61.00	78.25%
TAMP [ 17 ]	57.64	73.20	1571.36	39.67	37.65	89.54	78.58	69.64	61.00	86.56%
ATV-Pruning (Ours)	58.05	74.57	1406.75	40.11	46.15	89.78	80.47	70.23	61.98	88.07%

Appendix

Table A6 : Additional pruning results for LLaVA-OneVision and Qwen2.5-VL at 60% unstructured sparsity, serving as an extension of “Table 2”. Best results are highlighted in green . SQA

{}_{\text{img}}

results for Qwen2.5-VL are omitted because the current lmms-eval setup does not yield reliable evaluations.

Figure A6 : Trend of ATV-Pruning’s block-wise Visual Drift and Adaptive Token Allocation on LLaVA-NeXT (8B) with

\alpha=1.0

. The left y-axis measures the average visual drift per block. The right y-axis indicates the corresponding average number of visual tokens selected per sample. The trend illustrates the block-adaptive nature of our method: more tokens are retained in layers where visual representations undergo significant updates.

LVLM Pruning

See all 35 papers

IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models
Apr 1, 2026Dong-Jae Lee, Sunghyun Baek, Junmo KimEfficient VLM Inference Large Vision-Language Models
KAIST, South Korea
SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models
Aug 4, 2026Yaozhi Wen, Jialong Guo, Zhenliang Ni +2Efficient VLM Inference Structured Pruning
Huawei Technologies
ResPrune: Text-Conditioned Subspace Reconstruction for Visual Token Pruning in Large Vision-Language Models
Mar 22, 2026Xu Li, Yi Zheng, Yuxuan Liang +5Visual Token Pruning Efficient VLM Inference
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai 200433, China

Apr 1, 2026J/K move · Enter open · S save

IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models

Dong-Jae Lee, Sunghyun Baek, Junmo Kim

KAIST, South Korea

Efficient VLM Inference Large Vision-Language Models Multimodal Token Compression Attention Mechanisms Visual Token Pruning LVLM Pruning

Open paper AskPDF arXiv

Large Vision Language Models show impressive performance across image and video understanding tasks, yet their computational cost grows rapidly with the number of visual tokens. Existing token pruning methods mitigate this issue through empirical approaches while overlooking the internal mechanism of attention. In this paper, we propose a novel training free token pruning framework grounded in the dual form perspective of attention. We reformulate attention as an implicit linear layer whose weight matrix is the sum of rank 1 outer products, each generated by a single token's key value pair. Token pruning thus reduces to selecting an optimal subset of these rank 1 updates that best approximates the original dual weight matrix. Extending this perspective to standard softmax attention in LVLMs, we derive a novel metric quantifying both a token's information magnitude and information duplication. To efficiently select the subset with the proposed metric, we introduce Progressive Chunked Maximal Marginal Relevance. Extensive experiments demonstrate that our method achieves a better trade off between performance and efficiency, while providing another perspective on existing pruning approaches.

Abstract

LVLM Pruning

Explore similar work

IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models

SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models

ResPrune: Text-Conditioned Subspace Reconstruction for Visual Token Pruning in Large Vision-Language Models