Carve VFM: Carving Specialists out of Generalists, Automatic Task-Specific Compression of Vision Foundation Models

Lotfi Abdelkrim Mecharbat; Abdelrahman Elsayed; Ibrahim Almakky; Martin Takáč; Mohammad Yaqub

cs.CVOct 12, 2026

Carve VFM: Carving Specialists out of Generalists, Automatic Task-Specific Compression of Vision Foundation Models

Authors: Lotfi Abdelkrim Mecharbat, Abdelrahman Elsayed, Ibrahim Almakky, Martin Takáč, Mohammad Yaqub

Organizations: Mohamed bin Zayed University of Artificial Intelligence

Abstract

Vision foundation models (VFMs) provide transferable visual representations, but their inference cost limits deployment in many settings. Fine-tuning specializes a VFM without shrinking it; distillation shrinks it, but through expensive teacher-guided training. Structured pruning offers a more direct route by extracting a compact subnetwork from the VFM. However, as compression gets aggressive, existing methods fail because they treat subnetwork selection and recovery as separate axes. Across layers, capacity must follow task sensitivity rather than spread uniformly. Within a layer, a removal cannot be judged by importance alone; its impact also depends on how well the lost contribution can be recovered from what remains. Selection and recovery must therefore be decided jointly. We introduce Carve VFM, which resolves this interdependence. Given a VFM, a calibration set, and a parameter budget, it (i) searches for task-specific capacity allocations across layers; (ii) guides the search via an adaptive recovery-aware fitness criterion; and (iii) reconstructs removed behavior through iterative closed-form low-rank corrections merged into the surviving weights. The result is a dense encoder produced in minutes on a single GPU without backbone retraining. Across twelve benchmarks and four models from DINOv3 and MetaCLIP-2, Carve VFM achieves the highest reported mean accuracy in seven of eight backbone-budget configurations among the evaluated dense structured-pruning methods. Further fine-tuning surpasses the strongest evaluated edge model (88.1% vs. 85.8%), and carved models cut compute and memory while delivering 1.3-2.0x higher GPU throughput than the uncompressed one. These results show that the compact model you need is inside the VFM; it only has to be carved out.

Figures & tables

Figure 1: Task accuracy after pruning ViT-S from 21.6M to 10M parameters under different layer-wise allocations. Results are mean $\pm$ std over 3 seeds. The unpruned model is included for reference.

Figure 2: Overview of Carve VFM . From a VFM, a calibration set, and a parameter budget $B$ , the method (1) estimates removal damage, (2) searches layer-wise MLP retention and attention operators, and (3) carves and repairs the selected architecture. The output is a standard dense task-specific model within $B$ .

Figure 3: Structured pruning used in Carve VFM . MLP carving removes hidden channels with matching rows and columns from its two projections. Attention removes either complete heads (H) or some channels within each head (C).

Source	Budgets (M)			Repair rank
	Mild	Intermediate	Tight	$r$
DINOv3 ViT-S/16	14	10	6	128
DINOv3 ViT-B/16	60	35	20	128
MetaCLIP-2 ViT-B/16	60	35	20	256
DINOv3 ViT-L/16	200	150	100	256

Table 2: Source models, dense parameter budgets, and final repair rank.

Search setting	Value	Repair/linear-probing setting	Value
Population	20	Calibration examples	512
Elites	8	Ridge $\lambda$	$10^{-4}$
Generations	$\leq 12$	DINOv3 S/B rank	128
Candidate cap	120–200	MetaCLIP/ViT-L rank	256
Mutation $\sigma$	0.08	Linear-head epochs	10
Operator flip	0.18	Linear-head learning rate	$10^{-3}$

Table 8: Search, repair, and frozen-backbone linear-probing settings.

Source	Carve VFM	BlockDrop	FLAP	SVD-LLM	CORP
DINOv3 ViT-S/B	5.3 $\pm$ 2.1	1.5 $\pm$ 1.1	0.9 $\pm$ 0.5	1.4 $\pm$ 0.7	–
MetaCLIP-2 ViT-B	7.6 $\pm$ 1.8	2.0 $\pm$ 1.3	1.2 $\pm$ 0.6	2.1 $\pm$ 0.5	2.5 $\pm$ 1.6
DINOv3 ViT-L	16.4 $\pm$ 2.7	7.2 $\pm$ 2.9	2.2 $\pm$ 0.8	4.7 $\pm$ 0.8	27.1 $\pm$ 7.4

Table 9: Wall-clock minutes per complete task run, reported as mean

\pm

sample standard deviation. A dash marks the incomplete archived DINOv3 S/B CORP timing grid.

Model	Params (M)	General			Fine-grained				Texture/scene		Medical			Mean
		C10	C100	Cal.	Air.	CUB	Cars	Pets	DTD	MIT	Blood	Pneu.	OrganA
ResNet-18	11.2	93.7	77.0	82.0	61.1	69.6	63.1	90.4	64.3	68.7	98.7	90.5	95.4	79.6
ResNet-50	23.5	95.9	81.1	87.8	61.7	74.2	66.8	92.3	66.3	73.1	98.7	91.5	94.2	82.0
EfficientNet-B4	17.6	94.8	79.9	86.9	49.3	65.6	46.4	89.3	63.6	64.0	97.6	89.1	93.2	76.6
DeiT-Tiny	5.5	95.8	80.6	84.1	53.6	73.3	58.2	91.3	65.5	71.6	98.7	92.3	95.9	80.1
ConvNeXt-Pico	8.5	96.7	83.5	88.8	71.7	79.5	77.4	93.5	70.4	79.1	98.7	94.4	95.3	85.8

Table 10: Full-fine-tuning accuracy (%) under a 10-epoch schedule. Native models use their published pretrained weights; our rows fine-tune the already selected 14M and 10M compressed ViT-S children without rerunning structure search. All rows use batch size 32 and the same checkpoint-selection rule over epochs 1, 3, 5, and 10. Parameter counts vary slightly with the dataset-specific classifier; representative counts are shown. Bold denotes the best fully fine-tuned result in each column (ties at the reported precision are all highlighted). Accuracies are displayed to one decimal.

Device	Model	Budget (M)	GFLOPs $\downarrow$	Lat. (ms) $\downarrow$	img/s $\uparrow$	Mem. (MiB) $\downarrow$
CPU	ViT-L/16	Full	125.68	214.2 $\pm$ 1.8	7.4 $\pm$ 2.7	2246
	SVD-LLM	200	84.17	190.3 $\pm$ 1.6	9.3 $\pm$ 0.5	1837
	Carve VFM	200	83.49	117.4 $\pm$ 3.9	14.2 $\pm$ 0.6	1769 $\pm$ 18
	SVD-LLM	150	64.11	182.7 $\pm$ 1.3	9.8 $\pm$ 0.6	1641
	Carve VFM	150	63.30	105.6 $\pm$ 6.6	18.4 $\pm$ 0.7	1541 $\pm$ 13
	SVD-LLM	100	43.96	161.1 $\pm$ 1.0	10.4 $\pm$ 1.0	1453

Table 11: Deployment efficiency of ViT-L/16 compression at

224^{2}

resolution. We report compute, batch-1 latency, throughput, and peak memory usage on CPU and GPU. Throughput uses batch 16 on CPU and batch 64 on GPU; memory is peak process RSS on CPU and peak allocated device memory on GPU. Timings are mean

\pm

standard deviation.

Model Compression

Model compression reduces the storage, computation, or latency of machine-learning models while preserving task performance.

See all 122 papers

Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistency
Aug 28, 2026Jianjian Yin, Liulei Li, Tao Chen +3Neural Network Pruning Efficient ViTs
Nanjing University of Science and Technology, Nanjing, China · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery, Nanjing, China · Zhejiang University, Hangzhou, China. +1
Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency
Apr 27, 2026Yiran Huang, Lukas Thede, Massimiliano Mancini +2Large Vision-Language Models Structured Pruning
Technical University of Munich, Germany · Munich Center for Machine Learning, Germany. · Helmholtz Munich, Germany. +3
Who Speaks for the Pruned? Visual Token Pruning as Coverage Optimization
Sep 2, 2026Qingchan Zhu, Weihang You, Hanqi Jiang +3Efficient VLM Inference Visual Token Pruning
School of Computing, University of Georgia · College of Engineering, Northeastern University

Aug 28, 2026J/K move · Enter open · S save

Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistency

Jianjian Yin, Liulei Li, Tao Chen, Yi Chen+2

Nanjing University of Science and Technology, Nanjing, China · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery, Nanjing, China · Zhejiang University, Hangzhou, China.+1

Neural Network Pruning Efficient ViTs Vision Transformer

Open paper AskPDF arXiv

Pruning visual foundation models has attracted considerable attention. However, existing methods focus on rigid point-to-point token alignment on a single dataset for pruning, suffering from two limitations: i) robustness degradation, and ii) task-specificity deficiency. To address these limitations, we propose a task-specific pruning pipeline, named Cut-ViT. Specifically, we first construct gram anchoring matrices from both spatial and semantic perspectives, and perform the subspace decomposition to extract the corresponding subspace bases. Basis-agnostic and residual constraints are then adopted to align the gram subspaces between the native and pruned DINOv3 models along spatial and channel dimensions, enabling subnetworks to inherit robust feature representations of native DINOv3. Furthermore, we design spectral entropy adaptation, which quantifies the information density of feature manifolds along spatial and channel dimensions, thereby adapting the pruning objective to specific downstream tasks. Experiments show that Cut-ViT requires approximately one minute on a single A100 GPU to obtain subnetworks at various sparsity levels, using only 20.9% of the time and 45.5% of the GPU memory compared with previous methods, while achieving SOTA performance on six tasks across nine datasets.

Abstract

Model Compression

Explore similar work

Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistency

Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency

Who Speaks for the Pruned? Visual Token Pruning as Coverage Optimization