Understanding Pruning Regimes in Vision-Language Models Through Domain-Aware Layer Selection

Saeed Khaki; Nima Safaei; Kamal Ginotra

cs.CVMar 17, 2026

Understanding Pruning Regimes in Vision-Language Models Through Domain-Aware Layer Selection

Authors: Saeed Khaki, Nima Safaei, Kamal Ginotra

Organizations: Ohio State University

Abstract

Transformer-based vision-language models (VLMs) contain substantial depth redundancy, yet the effect of removing specific decoder layers remains poorly understood, especially for domains that require tight coupling between perception and multi-step reasoning. We study structured decoder layer pruning through the lens of domain-aware activation similarity, measuring how strongly each layer transforms representations for math versus non-math inputs. This yields simple math-aware, non-math-aware, and mixed ranking criteria that identify layers whose input-output activations change least within a target domain. Across two state-of-the-art VLMs and a broad suite of math and general multimodal benchmarks, we uncover a consistent three-regime structure: at low pruning budgets, performance is highly sensitive to which layers are removed; at moderate budgets, methods converge as structural damage accumulates; and at high budgets, structural continuity dominates, favoring spacing-aware strategies. Our domain-aware rankings achieve the strongest stability in the ranking-sensitive regime, while matching or exceeding structure-aware baselines at larger budgets. These results provide a clearer picture of how depth contributes to domain-specific behavior in VLMs and offer a practical, interpretable approach to reducing model depth without sacrificing essential mathematical or general vision-language capabilities.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the domain-aware decoder layer pruning pipeline. Domain-specific prompts drive activation logging; redundancy is estimated from layer in/out similarity and aggregated per domain. Layers are ranked (math-aware, non-math-aware, or mixed) and pruned under a fixed budget with protected endpoints, and a brief SFT stabilizes the pruned model with the vision encoder kept frozen.

Dataset	Size	Purpose
Snapask	300K total (100K train; 2,540 eval)	Real-world homework photos with CoT solutions; primary math supervision.
LLaVA-OneVision	105K total (100K train; 5K eval)	Broad multimodal instruction tuning: charts, diagrams, scenes, VQA.
NuminaMath-CoT	500 eval samples	High-difficulty math reasoning; domain-shift robustness.
ChartQA	32.7K questions (2,500 eval)	Reasoning over charts and structured data visualizations.
ScienceQA	21,208 total (2,017 eval)	Multimodal science questions with diagrams and multi-step reasoning.
RealWorldQA	765 images (full eval)	Real-world spatial and geometric reasoning.

Table 1: Training and evaluation datasets used in our experiments. Activation-analysis data is documented separately in Appendix A.1 .

Figure 2: Accuracy versus pruning budget across domains and model sizes. Shaded regions highlight the ranking-sensitive regime (0–15%), transition regime (15–32%), and structure-dominated regime (32–40%). Domain-aware methods separate most strongly at low budgets, while structural continuity becomes limiting at high budgets.

Figure 3: Trade-off between Math accuracy (x-axis) and General Vision-Language accuracy (y-axis) for Qwen3-VL-2B (left) and Qwen3-VL-4B (right). Each point represents a pruning method at a given budget; marker size increases with pruning budget (10%, 25%, 40%). The black star denotes the unpruned baseline. Domain-aware methods (circles) lie on or near the Pareto frontier, achieving the best balance between math and general VL performance. CKA and Random pruning are consistently dominated, while Interlace maintains general VL stability but exhibits steeper math degradation at higher budgets.

Figure 4: Decoder layer removal patterns across methods, budgets, and model sizes. Dark cells indicate pruned layers; light cells indicate retained layers. Domain-aware methods produce structured patterns, while Interlace enforces spaced deletions and Random remains unstructured.

Figure 5: Pruning candidate heatmaps for Qwen3-VL-2B (left) and Qwen3-VL-4B (right). Each cell shows the cosine similarity between a layer’s input and output activations for a given probing subtask. Higher similarity indicates weaker transformation and thus higher pruneability. Math subtasks (top rows) and non-math subtasks (bottom rows) exhibit distinct activation patterns, motivating domain-aware layer ranking.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	2B Base	2B Base+SFT	4B Base	4B Base+SFT
Snapask	40.47	32.83	50.75	44.96
NuminaMath	63.45	52.61	73.49	64.06
LLaVA-OneVision	59.12	62.00	67.26	68.00
ChartQA	79.72	32.60	84.00	35.12
RealWorldQA	65.23	66.93	71.50	72.29
ScienceQA	86.47	87.80	92.71	93.46

Appendix

Table 6: Reference accuracies for unpruned baselines. “Baseline” is the base model without fine-tuning. “Baseline+SFT” is the same model after the fixed SFT procedure used for post-pruning stabilization.

Figure 6: Relative accuracy drop from baseline by domain, model size, and pruning budget.

Figure 7: Per-benchmark accuracy curves for Qwen3-VL-2B.

Figure 8: Per-benchmark accuracy curves for Qwen3-VL-4B.

Figure 9: Best method per benchmark and pruning budget for 2B and 4B.

LVLM Pruning

See all 31 papers

Topology-Aware Layer Pruning for Large Vision-Language Models
Apr 14, 2026Pengcheng Zheng, Chaoning Zhang, Ya Wen +10LVLM Pruning Neural Network Pruning
University of Electronic Science and Technology of China · Kyung Hee University · Nota Inc. +2
Shallow to Deep: Aligning Token Pruning with Stage-wise Roles in LVLMs
Sep 22, 2026Shuo Zhang, Jintao Tong, Yixiong Zou +2Visual Token Pruning Efficient VLM Inference
School of Computer Science and Technology, Huazhong University of Science and Technology
From Patching to Pruning Visual Computation in Vision Language Models
Oct 2, 2026Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen +3Efficient VLM Inference Visual Token Pruning
Northeastern University, USA · EmbodyX Inc., USA · Zhejiang University, China +1

Apr 14, 2026J/K move · Enter open · S save

Topology-Aware Layer Pruning for Large Vision-Language Models

Pengcheng Zheng, Chaoning Zhang, Ya Wen, Wang Liu+9

University of Electronic Science and Technology of China · Kyung Hee University · Nota Inc.+2

LVLM Pruning Neural Network Pruning Large Vision-Language Models

Open paper AskPDF arXiv

Large Language Models (LLMs) have demonstrated strong capabilities in natural language understanding and reasoning, while recent extensions that incorporate visual inputs enable them to process multimodal information. Despite these advances, Large Vision-Language Models (LVLMs) incur substantial computational and memory costs, hindering deployment in resource-constrained scenarios. Existing layer pruning methods typically rely on local similarity metrics or static proxy signals, failing to capture the global and dynamic evolution of representations across model depth, which often leads to the removal of transition-critical layers. To address this limitation, we propose a topology-aware layer pruning framework for LVLMs. Specifically, we represent layer wise hidden states as point clouds and models their evolution using simplicial complexes. By leveraging zigzag persistent homology, we quantify inter-layer topological consistency and enable adaptive pruning that preserves critical representational transitions. Extensive experiments on diverse multimodal benchmarks demonstrate that the proposed framework consistently outperforms existing pruning methods across a wide range of sparsity ratios. Our code is available at https://github.com/zpc456/TopoVLM.

Abstract

LVLM Pruning

Explore similar work

Topology-Aware Layer Pruning for Large Vision-Language Models

Shallow to Deep: Aligning Token Pruning with Stage-wise Roles in LVLMs

From Patching to Pruning Visual Computation in Vision Language Models