Carve VFM: Carving Specialists out of Generalists, Automatic Task-Specific Compression of Vision Foundation Models

Lotfi Abdelkrim Mecharbat; Abdelrahman Elsayed; Ibrahim Almakky; Martin Takáč; Mohammad Yaqub

cs.CVOct 12, 2026

Carve VFM: Carving Specialists out of Generalists, Automatic Task-Specific Compression of Vision Foundation Models

Authors: Lotfi Abdelkrim Mecharbat, Abdelrahman Elsayed, Ibrahim Almakky, Martin Takáč, Mohammad Yaqub

Organizations: Mohamed bin Zayed University of Artificial Intelligence

Abstract

Vision foundation models (VFMs) provide transferable visual representations, but their inference cost limits deployment in many settings. Fine-tuning specializes a VFM without shrinking it; distillation shrinks it, but through expensive teacher-guided training. Structured pruning offers a more direct route by extracting a compact subnetwork from the VFM. However, as compression gets aggressive, existing methods fail because they treat subnetwork selection and recovery as separate axes. Across layers, capacity must follow task sensitivity rather than spread uniformly. Within a layer, a removal cannot be judged by importance alone; its impact also depends on how well the lost contribution can be recovered from what remains. Selection and recovery must therefore be decided jointly. We introduce Carve VFM, which resolves this interdependence. Given a VFM, a calibration set, and a parameter budget, it (i) searches for task-specific capacity allocations across layers; (ii) guides the search via an adaptive recovery-aware fitness criterion; and (iii) reconstructs removed behavior through iterative closed-form low-rank corrections merged into the surviving weights. The result is a dense encoder produced in minutes on a single GPU without backbone retraining. Across twelve benchmarks and four models from DINOv3 and MetaCLIP-2, Carve VFM achieves the highest reported mean accuracy in seven of eight backbone-budget configurations among the evaluated dense structured-pruning methods. Further fine-tuning surpasses the strongest evaluated edge model (88.1% vs. 85.8%), and carved models cut compute and memory while delivering 1.3-2.0x higher GPU throughput than the uncompressed one. These results show that the compact model you need is inside the VFM; it only has to be carved out.

Figures & tables

Figure 1: Task accuracy after pruning ViT-S from 21.6M to 10M parameters under different layer-wise allocations. Results are mean $\pm$ std over 3 seeds. The unpruned model is included for reference.

Figure 2: Overview of Carve VFM . From a VFM, a calibration set, and a parameter budget $B$ , the method (1) estimates removal damage, (2) searches layer-wise MLP retention and attention operators, and (3) carves and repairs the selected architecture. The output is a standard dense task-specific model within $B$ .

Figure 3: Structured pruning used in Carve VFM . MLP carving removes hidden channels with matching rows and columns from its two projections. Attention removes either complete heads (H) or some channels within each head (C).

Source	Budgets (M)			Repair rank
	Mild	Intermediate	Tight	$r$
DINOv3 ViT-S/16	14	10	6	128
DINOv3 ViT-B/16	60	35	20	128
MetaCLIP-2 ViT-B/16	60	35	20	256
DINOv3 ViT-L/16	200	150	100	256

Table 2: Source models, dense parameter budgets, and final repair rank.

Search setting	Value	Repair/linear-probing setting	Value
Population	20	Calibration examples	512
Elites	8	Ridge $\lambda$	$10^{-4}$
Generations	$\leq 12$	DINOv3 S/B rank	128
Candidate cap	120–200	MetaCLIP/ViT-L rank	256
Mutation $\sigma$	0.08	Linear-head epochs	10
Operator flip	0.18	Linear-head learning rate	$10^{-3}$

Table 8: Search, repair, and frozen-backbone linear-probing settings.

Source	Carve VFM	BlockDrop	FLAP	SVD-LLM	CORP
DINOv3 ViT-S/B	5.3 $\pm$ 2.1	1.5 $\pm$ 1.1	0.9 $\pm$ 0.5	1.4 $\pm$ 0.7	–
MetaCLIP-2 ViT-B	7.6 $\pm$ 1.8	2.0 $\pm$ 1.3	1.2 $\pm$ 0.6	2.1 $\pm$ 0.5	2.5 $\pm$ 1.6
DINOv3 ViT-L	16.4 $\pm$ 2.7	7.2 $\pm$ 2.9	2.2 $\pm$ 0.8	4.7 $\pm$ 0.8	27.1 $\pm$ 7.4

Table 9: Wall-clock minutes per complete task run, reported as mean

\pm

sample standard deviation. A dash marks the incomplete archived DINOv3 S/B CORP timing grid.

Model	Params (M)	General			Fine-grained				Texture/scene		Medical			Mean
		C10	C100	Cal.	Air.	CUB	Cars	Pets	DTD	MIT	Blood	Pneu.	OrganA
ResNet-18	11.2	93.7	77.0	82.0	61.1	69.6	63.1	90.4	64.3	68.7	98.7	90.5	95.4	79.6
ResNet-50	23.5	95.9	81.1	87.8	61.7	74.2	66.8	92.3	66.3	73.1	98.7	91.5	94.2	82.0
EfficientNet-B4	17.6	94.8	79.9	86.9	49.3	65.6	46.4	89.3	63.6	64.0	97.6	89.1	93.2	76.6
DeiT-Tiny	5.5	95.8	80.6	84.1	53.6	73.3	58.2	91.3	65.5	71.6	98.7	92.3	95.9	80.1
ConvNeXt-Pico	8.5	96.7	83.5	88.8	71.7	79.5	77.4	93.5	70.4	79.1	98.7	94.4	95.3	85.8

Table 10: Full-fine-tuning accuracy (%) under a 10-epoch schedule. Native models use their published pretrained weights; our rows fine-tune the already selected 14M and 10M compressed ViT-S children without rerunning structure search. All rows use batch size 32 and the same checkpoint-selection rule over epochs 1, 3, 5, and 10. Parameter counts vary slightly with the dataset-specific classifier; representative counts are shown. Bold denotes the best fully fine-tuned result in each column (ties at the reported precision are all highlighted). Accuracies are displayed to one decimal.

Device	Model	Budget (M)	GFLOPs $\downarrow$	Lat. (ms) $\downarrow$	img/s $\uparrow$	Mem. (MiB) $\downarrow$
CPU	ViT-L/16	Full	125.68	214.2 $\pm$ 1.8	7.4 $\pm$ 2.7	2246
	SVD-LLM	200	84.17	190.3 $\pm$ 1.6	9.3 $\pm$ 0.5	1837
	Carve VFM	200	83.49	117.4 $\pm$ 3.9	14.2 $\pm$ 0.6	1769 $\pm$ 18
	SVD-LLM	150	64.11	182.7 $\pm$ 1.3	9.8 $\pm$ 0.6	1641
	Carve VFM	150	63.30	105.6 $\pm$ 6.6	18.4 $\pm$ 0.7	1541 $\pm$ 13
	SVD-LLM	100	43.96	161.1 $\pm$ 1.0	10.4 $\pm$ 1.0	1453

Table 11: Deployment efficiency of ViT-L/16 compression at

224^{2}

resolution. We report compute, batch-1 latency, throughput, and peak memory usage on CPU and GPU. Throughput uses batch 16 on CPU and batch 64 on GPU; memory is peak process RSS on CPU and peak allocated device memory on GPU. Timings are mean

\pm

standard deviation.

Model Compression

Model compression reduces the storage, computation, or latency of machine-learning models while preserving task performance.

See all 122 papers

Aug 28, 2026

Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistency

Pruning visual foundation models has attracted considerable attention. However, existing methods focus on rigid point-to-point token alignment on a single dataset for pruning, suffering from two limitations: i) robustness degradation, and ii) task-specificity deficiency. To address these limitations, we propose a task-specific pruning pipeline, named Cut-ViT. Specifically, we first construct gram anchoring matrices from both spatial and semantic perspectives, and perform the subspace decomposition to extract the corresponding subspace bases. Basis-agnostic and residual constraints are then adopted to align the gram subspaces between the native and pruned DINOv3 models along spatial and channel dimensions, enabling subnetworks to inherit robust feature representations of native DINOv3. Furthermore, we design spectral entropy adaptation, which quantifies the information density of feature manifolds along spatial and channel dimensions, thereby adapting the pruning objective to specific downstream tasks. Experiments show that Cut-ViT requires approximately one minute on a single A100 GPU to obtain subnetworks at various sparsity levels, using only 20.9% of the time and 45.5% of the GPU memory compared with previous methods, while achieving SOTA performance on six tasks across nine datasets.

Jianjian Yin, Liulei Li, Tao Chen +3

Nanjing University of Science and Technology, Nanjing, China · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery, Nanjing, China · Zhejiang University, Hangzhou, China. +1

Neural Network Pruning Efficient ViTs Vision Transformer

Ask PDF

Apr 27, 2026

Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency

While Large Vision Language Models (LVLMs) demonstrate impressive capabilities, their substantial computational and memory requirements pose deployment challenges on resource-constrained edge devices. Current parameter reduction techniques primarily involve training LVLMs from small language models, but these methods offer limited flexibility and remain computationally intensive. We study a complementary route: compressing existing LVLMs by applying structured pruning to the language model backbone, followed by lightweight recovery training. Specifically, we investigate two structural pruning paradigms: layerwise and widthwise pruning, and pair them with supervised finetuning and knowledge distillation on logits and hidden states. Additionally, we assess the feasibility of conducting recovery training with only a small fraction of the available data. Our results show that widthwise pruning generally maintains better performance in low-resource scenarios, where computational resources are limited or there is insufficient finetuning data. As for the recovery training, finetuning only the multimodal projector is sufficient at small compression levels. Furthermore, a combination of supervised finetuning and hidden-state distillation yields optimal recovery across various pruning levels. Notably, effective recovery can be achieved using just 5% of the original data, while retaining over 95% of the original performance. Through empirical study on three representative LVLM families ranging from 3B to 7B parameters, this study offers actionable insights for practitioners to compress LVLMs without extensive computation resources or sufficient data. The code base is available at https://github.com/YiranHuangIrene/VLMCompression.git.

Yiran Huang, Lukas Thede, Massimiliano Mancini +2

Technical University of Munich, Germany · Munich Center for Machine Learning, Germany. · Helmholtz Munich, Germany. +3

Large Vision-Language Models Structured Pruning LVLM Pruning Data-Efficient Learning Neural Network Pruning

Ask PDF

Sep 2, 2026

Who Speaks for the Pruned? Visual Token Pruning as Coverage Optimization

Visual token pruning reduces the inference cost of vision-language models (VLMs), but most methods only ask which tokens to keep. This retained-token view can keep redundant high-scoring tokens while leaving discarded evidence without a close representative. We propose CoverPruner, a training-free pruner that asks the complementary demand-side question: after a token is removed, which surviving original token represents it for the target VLM? CoverPruner formulates pruning as Representational Coverage Maximization (RCM), covering the full projected visual-token set with query-weighted demand. It instantiates RCM with projector-space coverage and a lightweight first-layer attention probe. Across multiple VLM architectures and compression rates, CoverPruner achieves the best average accuracy among all compared methods, with the largest gains usually appearing under aggressive compression.

Qingchan Zhu, Weihang You, Hanqi Jiang +3

School of Computing, University of Georgia · College of Engineering, Northeastern University

Efficient VLM Inference Visual Token Pruning Visual Token Compression LVLM Pruning

Ask PDF

Abstract

Model Compression

Explore similar work

Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistency

Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency

Who Speaks for the Pruned? Visual Token Pruning as Coverage Optimization