HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Models

Lincen Bai; Hedi Tabia; Raul Santos-Rodriguez

cs.CVMar 6, 2026

HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Models

Authors: Lincen Bai, Hedi Tabia, Raul Santos-Rodriguez

Organizations: Université Paris-Saclay · University of Bristol

Abstract

Pruning vision-language models (VLMs) for efficient deployment is challenging because compression can affect not only task utility but also visual grounding, often amplifying object hallucinations even at the same sparsity level. We present HiPP-Prune, a hierarchical preference-conditioned structured pruning framework that treats pruning as conditional resource allocation under multiple objectives. HiPP-Prune makes plan-level decisions: a single policy invocation outputs a global pruning blueprint by factorizing decisions into an overall sparsity budget and a layer-wise allocation, enabling queryable trade-offs via a user-specified preference vector. To account for VLM-specific failure modes, our policy state integrates a visual sensitivity signal derived from attention flow between vision tokens and language hidden states, discouraging over-pruning of vision-critical layers that facilitate cross-modal fusion. We optimize pruning plans with plan-level Group Relative Policy Optimization (GRPO) under a multi-objective return that combines task utility, hallucination robustness (POPE), compression, and a synaptic-flow-inspired stability proxy to reduce unproductive exploration in high-sparsity regimes. Experiments on LLaVA with POPE and ScienceQA demonstrate that HiPP-Prune discovers diverse non-dominated pruning plans and provides controllable robustness--utility trade-offs under matched sparsity budgets.

Figures & tables

Figure 1: An overview of HiPP-Prune. A pretrained VLM and calibration data are used to construct layer-wise states, integrating activation statistics with vision-aware cross-modal cues. Conditioned on a preference vector, a hierarchical policy generates a one-shot structured pruning plan by factorizing decisions into global sparsity control and layer-wise allocation. Candidate plans are optimized via plan-level GRPO using robustness, utility, and compression feedback, while a SynFlow-inspired stability gate downweights updates from non-viable high-sparsity episodes to stabilize plan search. Finally, the optimized pruning plan is applied to the language backbone, followed by lightweight post-pruning recovery fine-tuning.

Method	Sparsity $\downarrow$	POPE BalAcc (%) $\uparrow$	SQA Acc (%) $\uparrow$
Dense (Unpruned)	0.000	82.43	64.00
Random	0.225	55.38 $\pm$ 1.99	32.83 $\pm$ 0.24
Wanda	0.225	51.14 $\pm$ 0.20	35.75 $\pm$ 1.25
LLM-Pruner	0.215	49.27 $\pm$ 1.06	35.50 $\pm$ 1.00
SliceGPT	0.229	52.56 $\pm$ 0.04	37.75 $\pm$ 0.25
HiPP-Prune (Ours)	0.225	72.89 $\pm$ 0.88	39.38 $\pm$ 0.74

Table 1: Main results on LLaVA-7B at matched sparsity ( 22.5%). POPE BalAcc and SQA Acc are reported as mean

\pm

std. For HiPP-Prune, we report the average across queried preference points.

Method	Sparsity $\approx 0.225$		Sparsity $\approx 0.325$
Method	POPE (%) $\uparrow$	SQA (%) $\uparrow$	POPE (%) $\uparrow$	SQA (%) $\uparrow$
Dense (Unpruned)	87.36	76.54	87.36	75.42
Random	52.77 $\pm$ 0.19	33.81 $\pm$ 3.06	50.86 $\pm$ 2.26	31.67 $\pm$ 2.10
Wanda	50.02 $\pm$ 1.98	34.36 $\pm$ 0.28	50.02 $\pm$ 1.98	29.73 $\pm$ 3.47
LLM-Pruner	53.69 $\pm$ 1.03	35.79 $\pm$ 0.43	50.39 $\pm$ 1.33	26.79 $\pm$ 0.57
SliceGPT	57.17 $\pm$ 0.19	36.15 $\pm$ 0.89	51.17 $\pm$ 0.19	31.15 $\pm$ 0.11

Table 2: Main results on Qwen2.5-VL-3B under two sparsity budgets. POPE BalAcc and SQA Acc are reported as mean

\pm

std over multiple runs; for HiPP-Prune, we report mean

\pm

std across queried preference points. All metrics are in percentages (%).

Preference $\mathbf{w}$	Sparsity $\downarrow$	POPE BalAcc (%) $\uparrow$	SQA Acc (%) $\uparrow$
$(0.60,0.30,0.10)$	0.2254	74.39	39.00
$(0.45,0.45,0.10)$	0.2254	72.56	39.50
$(0.33,0.33,0.34)$	0.2285	72.12	38.50
$(0.30,0.60,0.10)$	0.2253	72.48	40.50
mean $\pm$ std	0.2262 $\pm$ 0.0014	72.89 $\pm$ 0.88	39.38 $\pm$ 0.74

Table 3: Preference querying ablation on LLaVA-1.5-7B. Results are obtained by querying a single fixed policy with different preference weights. Sparsity is the realized structured ratio. POPE and SQA scores are reported in percentages (%). Bold indicates the highest score in each metric column.

Figure 2: Preference-conditioned operating-point clouds and Pareto navigation. Each point represents a pruning plan generated by a single trained HiPP-Prune policy under varying preference vectors $\mathbf{w}$ . Small dots illustrate the breadth of the search space by showing candidate plans sampled during the group-relative optimization process. Large dots denote the final Pareto-optimal blueprints selected by the policy for representative preference queries. Coloring reflects the sampled robustness weight $w_{\text{rob}}$ (ranging from 0 to 1), confirming that the policy effectively navigates the trade-off space between hallucination robustness and task utility by modulating the input vector $\mathbf{w}$ .

Training sampling	POPE (mean $\pm$ std) $\uparrow$	SQA (mean $\pm$ std) $\uparrow$
Full (anchor + Dirichlet)	76.62 $\pm$ 0.23	36.5 $\pm$ 0.42
w/o anchor (Dirichlet only)	76.01 $\pm$ 1.42	36.0 $\pm$ 0.58
w/o Dirichlet (anchor only)	76.34 $\pm$ 0.51	36.6 $\pm$ 1.04

Table 4: Preference sampling ablation under matched post-pruning recovery. Aggregated over four three-objective preference queries (balanced / robust-heavy / utility-heavy / compression-heavy).

LVLM Pruning

See all 31 papers

Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models
Aug 7, 2026Minseok Kang, Hyunwoo Kim, Chanyoung Kim +3Vision-Language Models Efficient VLM Inference
Department of Artificial Intelligence, Chung-Ang University, Seoul, Korea · School of Computer Science and Engineering, Soongsil University, Seoul, Korea · Department of Computer Science, Kookmin University, Seoul, Korea
GRIP-VLM: Group-Relative Importance Pruning for Efficient Vision-Language Models
May 13, 2026Mingzhe Huang, Weijun Wang, Xin Ding +7Efficient VLM Inference Visual Token Pruning
Institute for AI Industry Research (AIR), Tsinghua University · University of Science and Technology of China · Nanjing University +1
Collaborative Multi-Mode Pruning for Vision-Language Models
Apr 3, 2026Zimeng Wu, Yunhong Wang, Donghao Wang +1LVLM Pruning Efficient VLM Inference
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China

Aug 7, 2026J/K move · Enter open · S save

Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models

Minseok Kang, Hyunwoo Kim, Chanyoung Kim, Minwoo Kim+2

Department of Artificial Intelligence, Chung-Ang University, Seoul, Korea · School of Computer Science and Engineering, Soongsil University, Seoul, Korea · Department of Computer Science, Kookmin University, Seoul, Korea

Vision-Language Models Efficient VLM Inference LVLM Pruning Neural Network Pruning

Open paper AskPDF arXiv

Vision-language models (VLMs) have achieved remarkable generalization across diverse multimodal tasks through large-scale pre-training, yet their rapidly increasing computational and memory requirements pose significant challenges for deployment in constrained environments. Existing pruning strategies often depend on task-specific criteria or LLM-oriented importance measures, making them unsuitable for task-agnostic pruning, where no task-specific samples are available at pruning time and the pruned model remains broadly applicable. We introduce a retraining-free VLM pruning framework called PORTA that derives a task- and modality-agnostic importance formulation based on activation variation, estimated from generic calibration data, which reliably captures feature-level representation utility across modalities. PORTA further incorporates an adaptive sparsity allocation mechanism that assigns layer-wise pruning ratios based on output feature variability, avoiding the limitations of uniform sparsity and reducing performance degradation at high compression levels. Extensive experiments across VLM architectures, such as CLIP, BLIP, and Qwen2-VL, demonstrate that PORTA achieves competitive downstream performance under high sparsity without requiring any retraining, supporting efficient VLM compression. Code is available at https://github.com/cau-hai-lab/PORTA.git.

Abstract

LVLM Pruning

Explore similar work

Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models

GRIP-VLM: Group-Relative Importance Pruning for Efficient Vision-Language Models

Collaborative Multi-Mode Pruning for Vision-Language Models