Collaborative Multi-Mode Pruning for Vision-Language Models

Zimeng Wu; Yunhong Wang; Donghao Wang; Jiaxin Chen

cs.CVApr 3, 2026

Collaborative Multi-Mode Pruning for Vision-Language Models

Authors: Zimeng Wu, Yunhong Wang, Donghao Wang, Jiaxin Chen

Organizations: State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China

Abstract

Vision-Language Models (VLMs) have advanced rapidly within the unified Transformer architecture, yet their deployment on resource-constrained devices remains challenging due to high computational complexity. While pruning has emerged as an effective technique for compressing VLMs, existing approaches predominantly focus on a single mode by pruning either parameters or tokens, neglecting fully exploring the inherent redundancy in each mode, which leads to substantial performance degradation at high pruning ratios. To address the above limitations, we propose Collaborative Multi-Mode Pruning (CoMP), a novel framework tailored for VLMs by performing joint parameter and token pruning. Specifically, we first design a Collaborative Importance Metric (CIM) that investigates the mutual interference between the coupled parameters and tokens. It incorporates distinct significance of tokens into the computation of parameter importance scores, while simultaneously mitigating the affect of pruned parameters on token importance scores. Moreover, we develop a Multi-Mode Pruning Strategy (MPS) that decomposes the overall pruning process into a sequence of pruning stages, while in each stage we estimate the priory of different pruning modes based on their pruning cost and adaptively shift to the optimal one. Additionally, MPS integrates the historical cost and random exploration, in order to achieve a stable pruning process and avoid local optimum. Extensive experiments across various vision-language tasks and models demonstrate that our method effectively promotes the performance under high pruning ratios by comparing to the state-of-the-art approaches. The source code is available at https://github.com/Wuzimeng/CoMP.git.

Figures & tables

Figure 1: Illustration of different pruning modes for VLMs, with accuracy on NLVR2. For (i) parameter and (ii) token pruning, distinct modalities are simultaneously pruned under a unified ratio adjustment. For (iii) simple joint pruning, parameter and token pruning are conducted either sequentially or simultaneously without mitigating their inherent inconsistency. For (iv) our proposed CoMP, distinct pruning modes collaborate and only the optimal one is conducted at each stage in the progressive pruning process.

Panel of Figure 2 — Figure 2: (a) At $\mathit{layer}_{10}$ of BLIP’s vision encoder, tokens contributing most to parameter importance (blue marks) and those with top token importance are only slightly (lower than 30%) overlapped. (b) At $\mathit{layer}_{2}$ of BLIP’s vision encoder, 75% of the least important parameters (red boxes) still highly influence token importance.

Panel of Figure 3 — Figure 3: Framework overview of CoMP. (a) CoMP performs collaborative parameter and token pruning in nested loops. In the inner loop, input tokens are processed with partially masked parameters. The CIM module mitigates interference of progressive parameter pruning on token importance, and then suppresses the impact of redundant tokens for parameter importance. In the outer loop, the MPS module periodically selects the optimal pruning mode, whose corresponding threshold is adjusted to increase pruning ratio. (b) Given the full VLMs, CoMP compresses them by adaptively pruning parameters in different modalities, while enabling real-time token pruning during inference.

Panel of Figure 4 — Figure 4: Illustration of the CIM module. (a) adopts token-weighted input norm for parameter importance. (b) applies parameter pruning mask to the attention weight matrix for token importance.

Figure 5: Illustration of interference between parameter pruning and token importance. (a) Without pruning,

token_{1}

is more important than

token_{2}

. (b) Baseline pruning method masks redundant

head_{1}

by Eq. 2 , flattens softmax, and distorts ranks of token importance. (c) By masking with Eq. 7 ,

head_{1}

is gradually suppressed without disrupting correct ranks of token importance.

Method	Pruning Mode	Pruning Ratio	Dev. Acc.	Test Acc.	GFLOPs
Uncompressed	/	/	82.48	83.08	132.54
M-Pruning [ 42 ]	P	0.5	75.74	76.44	66.88
UPop [ 42 ]	P	0.5	76.89	77.61	65.29
STP [ 9 ]	T	0.5	78.08	77.61	68.31
MADTP [ 9 ]	T	0.5	81.75	82.35	66.36
CoMP (Ours)	C	0.5	81.67	82.61	66.74

Table 1: Comparison of Dev./Test Acc. (%) and GFLOPs by various pruning methods for BLIP on NLVR2 with distinct pruning ratios for the visual reasoning task. ‘P’, ‘T’, ‘J’ and ‘C’ denote parameter pruning, token pruning, joint pruning and collaborative pruning, respectively. ‘SJP’ stands for the simple joint pruning baseline. The best results are highlighted in bold .

Model	Method	Pruning Mode	Pruning Ratio	Flickr30K (1K test set)					COCO (5K test set)
				I $\rightarrow$ T		T $\rightarrow$ I		GFLOPs	I $\rightarrow$ T		T $\rightarrow$ I		GFLOPs
				R@1	R@5	R@1	R@5	GFLOPs	R@1	R@5	R@1	R@5	GFLOPs
BLIP	Uncompressed	/	/	96.8	99.9	86.9	97.3	91.65	81.9	95.4	64.3	85.7	91.65
	UPop * [ 42 ]	P	0.7	88.4	98.9	76.9	94.5	35.21	73.0	91.9	55.5	81.1	34.33
	MADTP * [ 9 ]	T	0.7	92.6	99.1	79.0	94.6	30.96	73.9	91.5	56.0	80.6	30.69
	CoMP (Ours)	C	0.7	94.4	99.6	80.1	95.1	30.47	76.2	92.4	57.0	81.1	30.08

Table 2: ( Top ) Comparison of R@1/5 (%) and GFLOPs for BLIP/CLIP on Flickr30K and COCO for the image-text retrieval task. ( Bottom ) Comparison of CIDEr and SPICE on COCO for the image captioning task and Test-dev/std accuracy (%) on VQAv2 for the visual question answering task as well as GFLOPs, based on BLIP. ‘*’ indicates GFLOPs during inference by our re-implementation, which were not reported in the original works. The best results are highlighted in bold .

Method	Pruning Mode	Pruning Ratio	MME	MMB	GQA	TextVQA	VQAv2	POPE	Average	TFLOPs
Uncommpressed	/	/	1862	64.7	61.9	58.2	78.5	85.9	69.28	5.63
FastV [ 11 ]	T	0.46	1612	61.2	52.7	52.5	67.1	64.8	59.31	3.18
SparseVLM [ 68 ]	T	0.46	1787	64.1	59.5	57.8	75.6	85.3	67.69	3.01
PDrop [ 60 ]	T	0.46	1797	63.3	57.3	56.5	75.1	82.3	66.45	3.09
VisionZip [ 63 ]	T	0.46	1783	63.0	59.3	57.3	76.8	85.3	67.56	3.01
DART [ 54 ]	T	0.46	1856	63.6	60.0	57.4	76.7	82.8	67.80	3.18

Table 3: Comparison of performance and TFLOPs by various pruning methods on LLaVA-v1.5-7B with distinct pruning ratios on

6

commonly-used benchmarks. The ‘Average’ column summarizes the average score across all tasks. ‘

\dagger

’ indicates that supervised fine-tuning is involved. The best results are highlighted in bold and the second-best results are underlined .

CIM	MPS	Dev. Acc.	Test Acc.	GFLOPs
		76.69	77.88	26.39
✓		78.20	78.60	26.04
✓	✓	79.23	79.62	25.97

Table 4: Ablation results of CIM and MPS on NLVR2 dataset at a pruning ratio of 0.8.

CIM	CIP	CIT		Dev. Acc.	Test Acc.	GFLOPs
				76.69	77.88	26.39
	✓			78.12	78.27	27.17
		✓		77.51	78.29	27.18
	✓	✓		78.20	78.60	26.04
MPS	CAS	RE	HI	Dev. Acc.	Test Acc.	GFLOPs
	✓			78.07	78.24	26.56

Table 5: ( Top ) Effect of CIP and CIT in CIM. CIP and CIT are token-weighted parameter importance and self-corrected token importance, respectively. ( Bottom ) Effect of CAS, RE and HI in MPS. CAS, RE and HI indicates pruning cost-aware mode shifting, random exploration and historical information, respectively. All experiments are conducted on NLVR2 at a pruning ratio of 0.8.

Figure 6: Visualization of pruning mode shifting in MPS with (w/) and without (w/o) priority refinement on NLVR2 at 0.8 pruning ratio. Colored points represent the next selected mode at each stage.

Configurations	BLIP-Reasoning	BLIP-Retrieval		BLIP-Captioning	BLIP-VQA	CLIP-Retrieval		LLaVA-Und
Configurations	NLVR2 [ 45 ]	COCO [ 32 ]	Flickr30K [ 65 ]	COCO [ 32 ]	VQAv2 [ 18 ]	COCO [ 32 ]	Flickr30K [ 65 ]	Mix665K [ 34 ]
Train batch size	32	64	16	64	16	16	16	32
Test batch size	64	64	32	64	16	32	32	1
Train epochs	30	10	18	9	2	10	10	1
Learning rate	3e-6	1e-7	1e-7	1e-5	5e-6	5e-6	5e-6	2e-5
Step size $\mathcal{S}_{P}$	0.02	0.02	0.02	0.02	-	0.01	0.01	0.0005

Table A : Training and testing configurations for pruning BLIP, CLIP and LLaVA models on various vision-language tasks. ‘Reasoning’, ‘Retrieval’, ‘Captioning’, ‘VQA’, ‘Und’ denote the visual reasoning task, the image-text retrieval task, the image captioning task, the visual question answering task and the image understanding task, respectively.

\text{R}_{\text{I}\rightarrow\text{T}}

and

\text{R}_{\text{T}\rightarrow\text{I}}

denote the recall for image-to-text and text-to-image retrieval, respectively.

-\text{Val\_Loss}

denotes the negative of the loss on validation set. ‘-’ indicates that the setting is not applied to the corresponding task.

Task	Input resolution	Vision Transformer					Language Transformer
Task	Input resolution	Number	Layers	Width	Heads	Intermediate	Number	Layers	Width	Heads	Intermediate
BLIP-Reasoning	384 $\times$ 384	2 $*$	12	768	12	3072	1	12	768	12	3072
BLIP-Retrieval	384 $\times$ 384	1	12	768	12	3072	1	12	768	12	3072
BLIP-Captioning	384 $\times$ 384	1	12	768	12	3072	1	12	768	12	3072
BLIP-VQA	480 $\times$ 480	1	12	768	12	3072	2	12	768	12	3072
CLIP-Retrieval	336 $\times$ 336	1	24	1024	16	4096	1	12	768	12	3072

Table B : Configurations of the models for multiple tasks.

*

indicates two images share one vision transformer in a single forward propagation during inference.

Figure A : Illustration of extending CoMP to the LLaVA-style architecture. We perform pruning in the LLM component, including pruning of vision tokens, language tokens and parameters, since this part accounts for more than 95% of the overall FLOPs.

Figure B : Illustration of the dimension mapping for parameter pruning in FFN (Left) and MHA (Right).

Figure C : FLOPs contributions of different pruning modes for the BLIP model, evaluated on the NLVR2 dataset at a pruning ratio of 0.85.

Method	GFLOPs	Latency $\downarrow$	Speedup Ratio $\uparrow$
Uncompressed	132.54	4.85	1.00 $\times$
UPop [ 42 ]	20.01	2.61	1.86 $\times$
MADTP [ 9 ]	20.57	1.27	3.82 $\times$
CoMP (Ours)	20.26	1.25	3.85 $\times$

Table C : Comparison of GFLOPs, Latency (ms per image) and Speedup Ratio across various pruning methods on BLIP using NLVR2 dataset at a pruning ratio of 0.85. Best results are highlighted in bold .

$\rho$	Dev. Acc.	Test Acc.	GFLOPs
0.1	78.70	79.08	26.00
0.2	79.23	79.62	25.97
0.3	79.50	79.76	25.95
0.4	78.79	79.55	26.68

Table D : Ablation results of the random exploration ratio

\rho

on BLIP using NLVR2 dataset at a pruning ratio of 0.8. The default setting is underlined .

Setting			Dev. Acc.	Test Acc.	GFLOPs
$\lambda_{0}$	$I_{\text{max}}$	Formulation of $\lambda$	Dev. Acc.	Test Acc.	GFLOPs
0.3	1	$\lambda\!=\!0.3$	79.07	79.03	26.51
0.4	1	$\lambda\!=\!0.4$	79.18	79.33	25.93
0.5	1	$\lambda\!=\!0.5$	78.75	79.02	25.91
0.4	2	$\lambda\!=\!\max(0.4\!-\!0.20(I_{m}\!-\!1),0)$	78.77	78.79	26.32
0.4	4	$\lambda\!=\!\max(0.4\!-\!0.10(I_{m}\!-\!1),0)$	79.35	79.45	26.31

Table E : Ablation results of the decay factor

\lambda

(parameterized by

\lambda_{0}

and

I_{\text{max}}

) on BLIP using NLVR2 dataset at a pruning ratio of 0.8. The default setting is underlined .

$\mathcal{I}$	Dev. Acc.	Test Acc.	GFLOPs
240	79.42	79.25	26.50
300	79.23	79.62	25.97
360	79.09	79.71	26.19

Table F : Ablation results of the interval steps

\mathcal{I}

on BLIP using the NLVR2 dataset at a pruning ratio of 0.8. The default setting is underlined .

$\tau$	Dev. Acc.	Test Acc.	GFLOPs
1	79.30	79.69	26.38
5	79.23	79.62	25.97
10	79.12	79.49	26.68

Table G : Ablation results of

\tau

in Eq. 9 on BLIP using the NLVR2 dataset at a pruning ratio of 0.8. The default setting is underlined .

Method	NLVR2-Reasoning			COCO-Retrieval
	Dev. Acc.	Test Acc.	GFLOPs	I $\rightarrow$ T		T $\rightarrow$ I		GFLOPs
	Dev. Acc.	Test Acc.	GFLOPs	R@1	R@5	R@1	R@5	GFLOPs
MADTP [ 9 ]	77.16 $\pm$ 0.57	77.64 $\pm$ 0.47	26.77 $\pm$ 0.23	74.38 $\pm$ 0.32	91.50 $\pm$ 0.24	56.28 $\pm$ 0.33	80.74 $\pm$ 0.13	30.04 $\pm$ 0.44
CoMP (Ours)	79.13 $\pm$ 0.24	79.67 $\pm$ 0.28	26.33 $\pm$ 0.38	75.96 $\pm$ 0.27	92.54 $\pm$ 0.13	57.32 $\pm$ 0.31	81.44 $\pm$ 0.30	29.24 $\pm$ 0.32

Table H : Comparison of Dev./Test Acc. (%), R@1/5 (%) and GFLOPs by CoMP and MADTP on BLIP for NLVR2 visual reasoning and COCO image-text retrieval tasks. Reported as mean

\pm

std over 5 seeds. The best results are highlighted in bold .

Method	Pruning Mode	Pruning Ratio	V $\rightarrow$ T		T $\rightarrow$ V		GFLOPs
Method	Pruning Mode	Pruning Ratio	R@1	R@5	R@1	R@5	GFLOPs
Uncompressed	/	/	35.8	59.7	43.3	65.5	733.4
UPop [ 42 ]	P	0.7	19.1	37.3	24.0	42.2	256.2
MADTP [ 9 ]	T	0.7	34.8	59.8	38.9	62.4	265.3
CoMP (Ours)	C	0.7	35.4	60.3	39.5	63.1	261.1

Table I : Comparison of R@1/5 (%) and GFLOPs for BLIP on MSR-VTT dataset for the zero-shot video-text retrieval task. The best results are highlighted in bold .

Figure D : Comparison of test accuracy with (w/) and without (w/o) the proposed MPS module on NLVR2 at an overall FLOPs pruning ratio of 0.8.

Method	Pruning Ratio	Dev. Acc.	Test Acc.	GFLOPs
CoMP w/ UPop [ 42 ]	0.8	79.23	79.62	25.97
CoMP w/ Isomorphic Pruning [ 15 ]	0.8	79.09	79.62	26.53
CoMP w/ UPop [ 42 ]	0.85	75.81	76.08	20.26
CoMP w/ Isomorphic Pruning [ 15 ]	0.85	77.23	77.75	19.83

Table J : Comparison of Dev./Test Acc. (%) and GFLOPs by CoMP with different parameter pruning frameworks on BLIP using NLVR2 dataset. The best results are highlighted in bold .

Figure E : Visualization of more examples illustrating inconsistency between parameter importance and token importance in the visual encoder of BLIP. Each example presents, from left to right: (1) the original image; (2) the remaining tokens after token pruning at $\mathit{layer}_{10}$ , along with the tokens that contribute most to parameter importance (blue regions), where the overlapping tokens are indicated by red boxes; (3) the heatmap of parameter contributions to token importance at $\mathit{layer}_{2}$ , with the least important parameters (to be pruned by parameter pruning) highlighted by red boxes.

LVLM Pruning

See all 31 papers

Aug 4, 2026

SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models

While Vision-Language Models (VLMs) have demonstrated remarkable performance in processing and understanding both text and images, their large parameter sizes lead to significant computational overhead, limiting their deployment on resource-constrained devices. While pruning has been effective for compressing Large Language Models (LLMs), directly applying it to VLMs leads to significant performance drops, largely due to redundant visual tokens interfering with importance estimation. To this end, we propose SlimVLM, a structured pruning framework designed to compress VLMs while preserving their task performance. We introduce an adaptive visual token selection strategy for VLMs that leverages average text-to-visual attention scores to assess the importance of visual tokens, removing redundant ones during pruning based on a set threshold, thereby optimizing the importance calculation. Recognizing the varying tolerance to sparsity across different modules, we also propose a Sensitivity-aware dynamic pruning mechanism that determines the appropriate pruning ratio for each module by calculating the linear reconstruction error between the outputs of the pruned and unpruned modules, ensuring overall performance stability. Experimental results show that SlimVLM outperforms existing methods across multiple multimodal benchmarks, achieving state-of-the-art performance.

Yaozhi Wen, Jialong Guo, Zhenliang Ni +2

Huawei Technologies

Efficient VLM Inference Structured Pruning Visual Token Pruning LVLM Pruning Neural Network Pruning

Ask PDF

Aug 7, 2026

Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models

Vision-language models (VLMs) have achieved remarkable generalization across diverse multimodal tasks through large-scale pre-training, yet their rapidly increasing computational and memory requirements pose significant challenges for deployment in constrained environments. Existing pruning strategies often depend on task-specific criteria or LLM-oriented importance measures, making them unsuitable for task-agnostic pruning, where no task-specific samples are available at pruning time and the pruned model remains broadly applicable. We introduce a retraining-free VLM pruning framework called PORTA that derives a task- and modality-agnostic importance formulation based on activation variation, estimated from generic calibration data, which reliably captures feature-level representation utility across modalities. PORTA further incorporates an adaptive sparsity allocation mechanism that assigns layer-wise pruning ratios based on output feature variability, avoiding the limitations of uniform sparsity and reducing performance degradation at high compression levels. Extensive experiments across VLM architectures, such as CLIP, BLIP, and Qwen2-VL, demonstrate that PORTA achieves competitive downstream performance under high sparsity without requiring any retraining, supporting efficient VLM compression. Code is available at https://github.com/cau-hai-lab/PORTA.git.

Minseok Kang, Hyunwoo Kim, Chanyoung Kim +3

Department of Artificial Intelligence, Chung-Ang University, Seoul, Korea · School of Computer Science and Engineering, Soongsil University, Seoul, Korea · Department of Computer Science, Kookmin University, Seoul, Korea

Vision-Language Models Efficient VLM Inference LVLM Pruning Neural Network Pruning

Ask PDF

May 13, 2026

GRIP-VLM: Group-Relative Importance Pruning for Efficient Vision-Language Models

In Vision-Language Models (VLMs), processing a massive number of visual tokens incurs prohibitive computational overhead. While recent training-aware pruning methods attempt to selectively discard redundant tokens, they largely rely on continuous-gradient relaxations. However, visual token pruning is inherently a discrete, non-convex combinatorial problem; consequently, these continuous approximations frequently trap the optimization in sub-optimal local minima, especially under aggressive compression budgets. To overcome this fundamental bottleneck, we propose GRIP-VLM, a Group-Relative Importance Pruning framework driven by Reinforcement Learning. Rather than relying on smooth-gradient assumptions, GRIP-VLM formulates pruning as a Markov Decision Process, employing a Group Relative Policy Optimization (GRPO) paradigm anchored by supervised warm-up to directly explore the discrete selection space. Integrated with a budget-aware scorer, our lightweight agent dynamically evaluates per-token importance and adapts to arbitrary compression ratios without retraining. Extensive experiments across diverse multimodal benchmarks demonstrate that GRIP-VLM consistently outperforms heuristic and supervised-learning baselines, achieving a superior Pareto frontier and delivering up to a 15% inference speedup at equal accuracy.

Mingzhe Huang, Weijun Wang, Xin Ding +7

Institute for AI Industry Research (AIR), Tsinghua University · University of Science and Technology of China · Nanjing University +1

Efficient VLM Inference Visual Token Pruning RL for VLMs

Ask PDF

Abstract

LVLM Pruning

Explore similar work

SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models

Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models

GRIP-VLM: Group-Relative Importance Pruning for Efficient Vision-Language Models