Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistency

Jianjian Yin; Liulei Li; Tao Chen; Yi Chen; Yazhou Yao; Wenguan Wang

cs.CVAug 28, 2026

Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistency

Authors: Jianjian Yin, Liulei Li, Tao Chen, Yi Chen, Yazhou Yao, Wenguan Wang

Organizations: Nanjing University of Science and Technology, Nanjing, China · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery, Nanjing, China · Zhejiang University, Hangzhou, China. · Nanjing Normal University, Nanjing, China

Abstract

Pruning visual foundation models has attracted considerable attention. However, existing methods focus on rigid point-to-point token alignment on a single dataset for pruning, suffering from two limitations: i) robustness degradation, and ii) task-specificity deficiency. To address these limitations, we propose a task-specific pruning pipeline, named Cut-ViT. Specifically, we first construct gram anchoring matrices from both spatial and semantic perspectives, and perform the subspace decomposition to extract the corresponding subspace bases. Basis-agnostic and residual constraints are then adopted to align the gram subspaces between the native and pruned DINOv3 models along spatial and channel dimensions, enabling subnetworks to inherit robust feature representations of native DINOv3. Furthermore, we design spectral entropy adaptation, which quantifies the information density of feature manifolds along spatial and channel dimensions, thereby adapting the pruning objective to specific downstream tasks. Experiments show that Cut-ViT requires approximately one minute on a single A100 GPU to obtain subnetworks at various sparsity levels, using only 20.9% of the time and 45.5% of the GPU memory compared with previous methods, while achieving SOTA performance on six tasks across nine datasets.

Figures & tables

Figure 1: Comparison with prior work. (a) To ensure feature consistency, existing methods enforce rigid alignment between CLS tokens; (b) Cut-ViT aligns the subspace induced by the inherent gram anchoring within DINOv3, thereby enabling the subnetworks to inherit the robust feature representations of the original network.

Figure 2: The pipeline of our Cut-ViT . Pairs of images are input into the native and pruning-guided DINOv3 to obtain feature embeddings. $\mathcal{L}_{ba}$ denotes the dense feature matching loss. Subspace decomposition ( $\lx@sectionsign\ref{GSC}$ ) is applied to the spatial and channel gram matrices ( $\bm{S}_{t}$ , $\bm{S}_{p}$ , $\bm{C}_{t}$ , $\bm{C}_{p}$ ) to derive subspace bases ( $\bm{U}_{t}$ , $\bm{U}_{p}$ ). Subspace consistency is enforced with basis-agnostic and residual constraints ( $\lx@sectionsign\ref{SAR}$ ), yielding spatial and channel gram losses, $\mathcal{L}^{\text{spatial}}_{\text{basis}}$ and $\mathcal{L}^{\text{channel}}_{\text{basis}}$ . During pruning, backpropagation computes the gradients of each parameter, which are then ranked in descending order of magnitude to select parameters and generate subnetworks with varying sparsity levels.

Figure 3: The process of spatial and channel gram anchoring subspace consistency.

		Different Sparsity Levels of DINOv3
Method	Dataset	30%	20%	10%
Training-based (0% : 69.7 / 67.00 / 72.5)
HydraViT [NeurIPS24] [ 16 ]	DAVIS	58.6 / 58.1 / 59.1	64.5 / 63.1 / 65.9	68.4 / 66.1 / 70.7
EA-ViT [ICCV25] [ 77 ]	DAVIS	60.3 / 59.4 / 61.2	65.4 / 63.9 / 66.9	69.3 / 66.8 / 71.8
Training-free (0% : 69.7 / 67.0 / 72.5)
LAMP [ICLR21] [ 25 ]	ImageNet	35.0 / 32.9 / 37.0	40.4 / 38.1 / 42.8	60.2 / 58.4 / 62.1

Table 1: Quantitative results (

\lx@sectionsign\ref{compare}

) on DAVIS-2017 [ 39 ] video object segmentation using

(\mathcal{J}\&\mathcal{F})_{m}

\mathcal{J}_{m}

\mathcal{F}_{m}

		Different Sparsity Levels of DINOv3
Method	Dataset	30%	20%	10%
Training-based (0% : 92.0 / 77.1 / 59.7)
HydraViT [NeurIPS24] [ 16 ]	FJS	69.9 / 55.7 / 37.6	90.7 / 72.5 / 51.9	91.2 / 76.3 / 59.1
EA-ViT [ICCV25] [ 77 ]	FJS	72.0 / 57.9 / 38.5	91.4 / 74.2 / 54.6	92.0 / 77.0 / 59.5
Training-free (0% : 92.0 / 77.1 / 59.7)
LAMP [ICLR21] [ 25 ]	ImageNet	55.2 / 38.8 / 29.9	79.5 / 56.9 / 38.9	84.2 / 59.2 / 50.4

Table 2: Quantitative results (

\lx@sectionsign\ref{compare}

) of semantic matching on the FG3DCar / JODS / SBD (FJS) dataset [ 33 , 44 , 17 ] .

Table 6

Table 7

Table 8

Table 9

Table 10

		Video Object Segmentation	Depth Estimation	Semantic Matching
Method	Dataset	$(\mathcal{J}\&\mathcal{F})_{m}$ / $\mathcal{J}_{m}$ / $\mathcal{F}_{m}$	ARel $\downarrow$ / $\delta_{1}$ $\uparrow$	FG3DCar / JODS / SBD
	ImageNet	52.9 / 50.7 / 55.1	8.3 / 97.6	64.8 / 44.2 / 32.1
SNOWS [ICLR25] [ 35 ]	Target	53.4 / 50.9 / 56.0	8.0 / 97.6	65.4 / 44.6 / 32.6
	ImageNet	56.0 / 54.3 / 57.7	7.7 / 97.6	65.7 / 43.3 /32.4
SnapViT [NeurIPS25] [ 50 ]	Target	56.4 / 54.6 / 58.1	7.3 / 97.7	66.4 / 43.5 / 32.8
Cut-ViT (Ours)	Target	59.4 / 58.0 / 60.8 $\uparrow$ 3.0 / $\uparrow$ 3.4 / $\uparrow$ 2.7	4.6 / 98.7 $\uparrow$ 2.7 / $\uparrow$ 1.0	70.8 / 56.3 / 37.9 $\uparrow$ 4.4 / $\uparrow$ 11.7 / $\uparrow$ 5.1

Table 13: Ablation analysis (

\lx@sectionsign\ref{ablation_study}

) on different pruning datasets at 30% sparsity.

Table 12

Figure 4: Qualitative comparison ( $\lx@sectionsign\ref{ablation_study}$ ) of different subspace alignment strategies.

Figure 5: The parameter retention and overlap analysis (

\lx@sectionsign\ref{ablation_study}

) for each block in DINOv3 [ 49 ] . ( Left ) shows the parameter retention rate for each block in both training-free OSP and our task-specific methods. ( Right ) illustrates the overlap ratio of the retained parameters in the MLP and MHA layers of each block for the two types of methods.

Neural Network Pruning

Neural network pruning removes weights, neurons, channels, heads, or layers to reduce model size and computation while retaining useful behavior.

See all 292 papers

Carve VFM: Carving Specialists out of Generalists, Automatic Task-Specific Compression of Vision Foundation Models
Oct 12, 2026Lotfi Abdelkrim Mecharbat, Abdelrahman Elsayed, Ibrahim Almakky +2Model Compression Neural Network Pruning
Mohamed bin Zayed University of Artificial Intelligence
Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models
Aug 7, 2026Minseok Kang, Hyunwoo Kim, Chanyoung Kim +3Vision-Language Models Efficient VLM Inference
Department of Artificial Intelligence, Chung-Ang University, Seoul, Korea · School of Computer Science and Engineering, Soongsil University, Seoul, Korea · Department of Computer Science, Kookmin University, Seoul, Korea
LearnPrunerV2: Learnable Progressive Token Pruning for Vision-Language Models
Oct 27, 2026Rinyoichi Takezoe, Yaqian Li, Weiren Wang +6Visual Token Pruning LVLM Pruning
Li Auto Inc. Beijing, China · Li Auto Inc. Shanghai, China · Li Auto Inc. Hangzhou, China

Oct 12, 2026J/K move · Enter open · S save

Carve VFM: Carving Specialists out of Generalists, Automatic Task-Specific Compression of Vision Foundation Models

Lotfi Abdelkrim Mecharbat, Abdelrahman Elsayed, Ibrahim Almakky, Martin Takáč+1

Mohamed bin Zayed University of Artificial Intelligence

Model Compression Neural Network Pruning Structured Pruning Vision Foundation Models

Open paper AskPDF arXiv

Vision foundation models (VFMs) provide transferable visual representations, but their inference cost limits deployment in many settings. Fine-tuning specializes a VFM without shrinking it; distillation shrinks it, but through expensive teacher-guided training. Structured pruning offers a more direct route by extracting a compact subnetwork from the VFM. However, as compression gets aggressive, existing methods fail because they treat subnetwork selection and recovery as separate axes. Across layers, capacity must follow task sensitivity rather than spread uniformly. Within a layer, a removal cannot be judged by importance alone; its impact also depends on how well the lost contribution can be recovered from what remains. Selection and recovery must therefore be decided jointly. We introduce Carve VFM, which resolves this interdependence. Given a VFM, a calibration set, and a parameter budget, it (i) searches for task-specific capacity allocations across layers; (ii) guides the search via an adaptive recovery-aware fitness criterion; and (iii) reconstructs removed behavior through iterative closed-form low-rank corrections merged into the surviving weights. The result is a dense encoder produced in minutes on a single GPU without backbone retraining. Across twelve benchmarks and four models from DINOv3 and MetaCLIP-2, Carve VFM achieves the highest reported mean accuracy in seven of eight backbone-budget configurations among the evaluated dense structured-pruning methods. Further fine-tuning surpasses the strongest evaluated edge model (88.1% vs. 85.8%), and carved models cut compute and memory while delivering 1.3-2.0x higher GPU throughput than the uncompressed one. These results show that the compact model you need is inside the VFM; it only has to be carved out.

Abstract

Neural Network Pruning

Explore similar work

Carve VFM: Carving Specialists out of Generalists, Automatic Task-Specific Compression of Vision Foundation Models

Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models

LearnPrunerV2: Learnable Progressive Token Pruning for Vision-Language Models