In this paper, we present Chain-of-Models Pre-Training (CoM-PT), a novel performance-lossless training acceleration method for vision foundation models (VFMs). This approach fundamentally differs from existing acceleration methods in its core motivation: rather than optimizing each model individually, CoM-PT is designed to accelerate the training pipeline at the model family level, scaling efficiently as the model family expands. Specifically, CoM-PT establishes a pre-training sequence for the model family, arranged in ascending order of model size, called model chain. In this chain, only the smallest model undergoes standard individual pre-training, while the other models are efficiently trained through sequential inverse knowledge transfer from their smaller predecessors by jointly reusing the knowledge in the parameter space and the feature space. As a result, CoM-PT enables all models to achieve performance that is mostly superior to standard individual training while significantly reducing training cost, and this is extensively validated across 45 datasets spanning zero-shot and fine-tuning tasks. Notably, its efficient scaling property yields a remarkable phenomenon: training more models even results in higher efficiency. For instance, when pre-training on CC3M: i) given ViT-L as the largest model, progressively prepending smaller models to the model chain reduces computational complexity by up to 72%; ii) within a fixed model size range, as the VFM family scales across 3, 4, and 7 models, the acceleration ratio of CoM-PT exhibits a striking leap: from 4.13X to 5.68X and 7.09X. Since CoM-PT is naturally agnostic to specific pre-training paradigms, we open-source the code to spur further extensions in more computationally intensive scenarios, such as large language model pre-training.
Figures & tables
Figure 1: Standard individual pre-training vs . Chain-of-Models Pre-Training . All ViTs [ 17 ] are trained on a combined dataset of CC3M [ 59 ] and CC12M [ 4 ] , and evaluated on ImageNet-1K [ 14 ] .
Figure 2: Overview of Chain-of-Models Pre-training pipeline with inverse knowledge transfer relay. Models are organized sequentially by ascending model size, from the smallest model m1 to the largest mn , forming a model chain. m1 first undergoes standard individual pre-training. Each subsequent mi is then efficiently trained via inverse knowledge transfer (denoted as \mathrel{\hbox to9.9pt{\vbox to7.2pt{\pgfpicture\makeatletter\hbox{\hskip 4.23409pt\lower-3.59999pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {{{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#FFFFFF} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.9pt} \lx@inpgf@ignorespaces{{}{}{}{}{}{}{}{}{{}{}}{}{{}{}\lx@inpgf@ignorespaces}{{}{}\lx@inpgf@ignorespaces}{{{}{}}}{{}{}\lx@inpgf@ignorespaces}{{{}{}}}{{}{}}{{}{}\lx@inpgf@ignorespaces}{{}{}\lx@inpgf@ignorespaces}{{}{}\lx@inpgf@ignorespaces}{}{}{}{{}{}\lx@inpgf@ignorespaces}{{}{}\lx@inpgf@ignorespaces}{{}{}\lx@inpgf@ignorespaces}{}{}{}{{}{}\lx@inpgf@ignorespaces}{{}{}\lx@inpgf@ignorespaces}{{}{}\lx@inpgf@ignorespaces}{}{}{}{}{}{}{}{}{}{}{{}}{}{{{}}{{\lx@inpgf@ignorespaces}}{}{}{}{}{}{}{}}{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#FFFFFF} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.9pt} \lx@inpgf@ignorespaces{}\lxSVG@fillstroke\lxSVG@drawpath@unclipped{M 7.22 0 L 2.86 4.36 L 2.86 1.98 L -5.24 1.98 L -5.24 -1.98 L 2.86 -1.98 L 2.86 -4.36 Z}{} \lx@inpgf@ignorespaces \lxSVG@closescope }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{58}\lxSVG@closescope }}} \lxSVG@closescope }}} } \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}} ) from its immediate predecessor mi−1 , forming a continuous relay along the model chain. Details of our proposed inverse knowledge transfer are in Figure 3 .
Figure 3: Inverse knowledge transfer from mi to mi+1 .
Figure 4: Convergence analysis across different training epochs. We train ViT-T/16, ViT-S/16, and ViT-B/16 on CC3M for 32, 64, 128, and 256 epochs to determine convergence points. Top-1 (%) represents zero-shot classification accuracy on ImageNet-1K.
Method
Text Augmentation Sources
# Epochs
Top-1(%)
CLIP
-
32
16.00
LaCLIP
LLaMA [ 64 ]
32
21.50
MLLM-A
Multi-MLLMs [ 83 , 32 , 2 , 40 ]
32
25.00
Our Baseline
ShareGPT4V [ 7 ]
32
28.80
(LaCLIP)
128
31.80
Table 1: Performance comparison with recent top methods on the CC3M dataset. Top-1 (%) represents zero-shot classification accuracy on ImageNet-1K. Our method achieves similar acceleration with baseline using 32 epochs, demonstrated in Figure 9(a) .
Model Family
Method
Model
Zero-shot Top-1/R@1 (%)
Training MACs ( 1010 G)
Training Time ( 102 GPU hrs)
ImageNet-1K
VTAB+
COCO
Individual
Accumulated
Individual
Accumulated
Pre-training Dataset: CC3M
ViT Family
Baseline
ViT-T/16
26.16
25.61
29.06
0.62
0.62
1.54
1.54
ViT-S/16
30.16
28.84
32.81
1.54
2.16
2.40
3.94
ViT-B/16
31.80
30.08
34.93
3.75
5.91
3.45
7.39
ViT-L/16
33.77
30.39
36.53
10.79
16.70
7.56
14.95
Table 2: Main results. Evaluation is conducted on three benchmarks: ImageNet-1K, VTAB+, and COCO. Efficiency is measured in terms of training MACs and training time. Specifically, training MACs include all computations in the forward and backward propagations, as well as parameter updates, and training time denotes the required NVIDIA A100 (80GB) GPU hours for pre-training. ‘Individual’ and ‘accumulated’ refer to specific model overhead and cumulative overhead along the model chain, respectively. Best results are bolded .
Figure 5: Comparison of acceleration ratios, measured by two efficiency metrics. We report both training MACs and A100 GPU hours for the ViT family pre-trained on the CC3M dataset.
Open-Vocabulary Semantic Segmentation Tasks
Vision-Language Tasks
Backbone
PT Method
ADE-847
ADE-150
PC-459
PC-59
VOC
Avg.
TextVQA
ScienceQA
POPE
VQAv2
Avg.
ViT-B/16
Baseline
0.45
11.40
1.07
36.12
68.59
23.53
43.08
62.57
73.62
54.16
58.36
CoM-PT
3.18
18.30
6.68
40.73
76.96
29.17
43.21
63.41
74.17
54.59
58.85
ViT-L/16
Baseline
1.03
13.58
4.47
33.99
69.76
24.57
43.64
63.56
74.50
55.68
59.35
CoM-PT
4.73
20.14
7.00
42.17
79.52
30.71
43.83
63.51
75.15
56.07
59.64
Table 3: Fine-tuning on downstream tasks. With backbones pre-trained by baseline method and CoM-PT, the framework SAN [ 70 ] for open-vocabulary semantic segmentation is fine-tuned on the COCO-Stuff [ 3 ] training set, while the LLaVA-1.5-7B [ 40 ] for vision-language tasks undergoes LoRA-based visual instruction fine-tuning following its official protocol. Both ViT-B/16 and ViT-L/16 are pre-trained on the CC3M dataset. The ‘PT method’ refers to the pre-training method of the backbone. Best results are bolded .
Figure 6: Reduced training MACs vs. model chain starting with smaller models. The reduced MACs represent the total training cost of the model chain relative to the cost of individually pre-training ViT-L. All models are performance-lossless.
Figure 7: Training MACs and acceleration ratios of model chains with different expansion ratios . Experiments are conducted on the CC3M dataset. All models are performance-lossless.
Figure 8: Training epoch allocation. Along model chain structured ViT-T → ViT-S → ViT-B → ViT-L, we determine the minimum training epochs required per model to achieve lossless acceleration via CoM-PT, compared to standard 128-epoch baselines on CC3M.
Table 4: Ablation Studies. (a) We study efficiency gain from the model chain, by comparing models progressively trained along the model chain (i.e., ViT-T → ViT-S → ViT-B) against those independently initialized from a trained ViT-T. (b) We study the complementarity of inverse weight initialization (IWI) and inverse feature distillation (IFD) using ViT-T → ViT-S short chain. All models are trained on CC3M.
Figure 9: Acceleration ratio variation under different training epochs and data scales. Experiments use the ViT family. (a) On CC3M, we vary the training epochs (32, 64, and 128). (b) We test the data scale from 22.0M to 206.8M under a fixed 64-epoch setting to isolate data scale effects. The curve fitting format follows [ 77 ] .
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
# Total Pairs
# Collected Pairs
# Augmented Pairs
CC3M
3,318,333
2,753,427
44,054,832
CC12M
12,423,374
10,173,631
162,778,096
Merged-15M
-
12,927,058
206,832,928
Appendix
Table A : CC3M, CC12M and our Merged-15M datasets used for pre-training in this paper. Note that we only report the number of samples in the training set.
Dataset
Abbr.
Test size
ImageNet-1K
-
50,000
ImageNet-v2
-
10,000
ImageNet-R
-
30,000
ImageNet Sketch
-
50,889
ObjectNet
-
18,574
ImageNet-A
-
7,500
Appendix
Table B : Datasets in VTAB+ with abbreviations and test sizes.
Table C : Detailed settings of baseline pre-training and chain-of-models pre-training on CC3M and Merged-15M datasets.
Dataset
ViT Family
Swin Family
ViT-T/16
ViT-S/16
ViT-M/16
ViT-B/16
ViT-L/16
Swin-T
Swin-S
Swin-B
Swin-L
CC3M
128
24
–
18
15
128
24
16
12
Merged-15M
–
64
21
17
15
–
–
–
–
Appendix
Table D: Training epochs for different models in main experiments.
Table E : Detailed settings of fine-tuning on COCO-Stuff for open-vocabulary semantic segmentation tasks.
Table F : Detailed settings of feature alignment training and LoRA-based visual instruction fine-tuning.
Model Family
Model
Image Encoder
Text Encoder
Overall
Width
Depth
Param (M)
Width
Depth
Param (M)
Param (M)
Forward MACs (G)
Training MACs (G)
ViT Family
ViT-T/16
192
12
5.62
256
12
9.48
15.10
5.82
17.57
ViT-C/16
256
12
9.86
320
12
14.80
24.66
9.23
27.87
ViT-S/16
384
12
21.81
384
12
21.29
43.10
14.44
43.65
ViT-M/16
512
12
38.59
448
12
28.97
67.56
21.88
66.17
ViT-B/16
768
12
86.19
512
12
37.83
124.02
35.09
106.27
Appendix
Table G : Architecture specifications for VFM families. Forward MACs and Training MACs represent the computational complexity of the model for a sample (an image and its corresponding 4 text descriptions) in our experiment.
Figure A : Detailed illustrations of weight initialization in width and depth. First, (a)-(c) illustrates different weight initialization approaches for width differences. Duplication (a) involves copying the teacher’s weights multiple times and directly assigning these copies to the student. Interpolation (b) conducts bilinear operations to expand the teacher’s weights to fit the shape of the student. Insertion (c) involves directly assigning the teacher’s weights to the student, with all remaining weights initialized randomly. Second, (d)-(f) show the different weight initialization methods for depth differences. (d) only initializes the first p layers across the total q layers, while the interval insertion (e) skips several layers by calculating q/p . As a combination, the duplicate insertion (f) initializes the skipped layers by duplicating the weights of the preceding layer.
Figure B : Ablation study on various simple weight initialization approaches. For width initialization, experiments are conducted on the ViT-T/16 → ViT-S/16 sub-chain. Progressively, with the best simple width initialization method, we explore depth initialization on the ViT-B/16 → ViT-L/16 sub-chain. Inverse feature distillation is applied as default. Details for each method are in Figure A .
Method
ZS Performance
ImageNet-1K
COCO
Baseline
26.48
28.34
+IFD
29.29
32.10
+IFD+IWI
30.24
34.15
+IFD+Net2Net [ 9 ]
30.07
33.63
+IFD+NetExpand [ 16 ]
30.36
34.13
Appendix
Table H : Our simple approaches vs. previous sophisticated designs. Experiments are conducted using the ViT-T/16 → ViT-S/16 sub-chain, trained on the CC3M dataset.
Figure C : Validation curves of different choices in Table 4(b) of the main paper.
Figure D : Ablation study on inverse feature distillation magnitude. For better illustration, we define r as the ratio between the magnitude of LIFD and Ltask . The experiment is conducted on ViT-T/16 → ViT-S/16 on the CC3M dataset. We show how IFD enables ViT-S/16 to achieve target performance (128-epoch individual pre-training) with only 24 epochs. The teacher model is ViT-T/16 trained individually for 128 epochs. Inverse weight initialization is applied as default.
PT Dataset
CC3M
Merged-15M
Models
ViT-T/16
ViT-S/16
ViT-B/16
ViT-L/16
Swin-T
Swin-S
Swin-B
Swin-L
ViT-S/16
ViT-M/16
ViT-B/16
ViT-L/16
PT Method
baseline
baseline
CoM-PT
baseline
CoM-PT
baseline
CoM-PT
baseline
baseline
CoM-PT
baseline
CoM-PT
baseline
CoM-PT
baseline
baseline
CoM-PT
baseline
CoM-PT
baseline
CoM-PT
ImageNet-1K
26.16
30.16
30.24
31.80
31.83
33.77
34.27
33.84
35.88
36.19
36.13
36.71
36.77
37.06
43.97
45.90
45.85
47.15
47.39
48.80
49.47
ImageNet-v2
22.85
26.47
25.91
27.42
28.00
29.39
29.94
29.00
30.99
31.31
31.54
31.59
31.83
32.30
38.24
39.52
39.85
41.35
40.79
43.06
42.84
ImageNet-R
37.15
42.36
40.77
44.09
43.76
47.60
47.94
47.35
50.70
50.72
52.16
52.04
51.85
52.96
63.28
66.06
65.61
69.16
67.70
71.65
71.02
ImageNet Sketch
16.93
21.24
20.53
22.81
21.32
24.54
24.76
24.92
26.25
26.80
27.66
27.12
27.42
27.52
35.09
36.74
36.96
39.27
38.69
41.06
41.10
Appendix
Table I : Zero-shot Top-1 accuracy (%) across the 35 datasets of the VTAB+ benchmark. Detailed results corresponding to the VTAB+ averages presented in Table 2 of the main paper.
Pretrained vision foundation models deliver strong performance across tasks with limited fine-tuning. However, their Vision Transformer (ViT) backbones impose high inference costs, limiting deployment on resource-constrained devices. In this work, we accelerate large-scale pretrained ViTs while preserving their feature extraction capabilities by exploiting the intrinsic convolution-like behavior of some attention heads. Specifically, we introduce an efficient depthwise convolution-based layer that serves as a drop-in replacement for these heads. Additionally, we propose simple strategies to identify which heads can be replaced and introduce a fine-tuning procedure that recovers downstream task performance. Across both image classification and segmentation tasks, our method achieves 17-20% percent inference speedup with minimal performance degradation. We validate the approach through detailed derivations, extensive experiments, and efficiency benchmarks. The reference implementation is publicly available.
Carmelo Scribano, Mohammad Mahdi, Nedyalko Prisadnikov +5
University of Modena and Reggio Emilia, Modena, Italy. · INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria
Adapting foundation models to a new task within or beyond the training domain is the de facto standard for building specialized machine learning models. Recently, Visual Prompting (VP) has emerged as a promising technique for efficient knowledge transfer. As new foundation model families~(like Mamba) get introduced and VP pipelines such as AutoVP reach greater maturity, we find a growing need for a systematic evaluation of current approaches. In this work, we assess the performance of the latest models, comparing them to earlier architectures and alternative fine-tuning methods, to better understand the progress, challenges and opportunities in the field of efficient fine-tuning under resource limitations. Towards this goal, this paper provides a concise empirical overview of the interactions among foundation model families (Attention-, Convolution-, and Mamba-based) and transfer paradigms: VP, Linear Probing (LP), Full Fine-tuning (FFT) and Low Rank Adaptation (LoRA). Our work builds up on previous findings by broadening the selection of evaluated models and techniques while tuning hyperparameters. In the interest of delivering practical guidelines for the user, we also explore application of prevalent regularization techniques to boost performance in the context of VP.
Niclas Hergenröther, Antonio Orvieto
ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center
Vision foundation models (VFMs) provide transferable visual representations, but their inference cost limits deployment in many settings. Fine-tuning specializes a VFM without shrinking it; distillation shrinks it, but through expensive teacher-guided training. Structured pruning offers a more direct route by extracting a compact subnetwork from the VFM. However, as compression gets aggressive, existing methods fail because they treat subnetwork selection and recovery as separate axes. Across layers, capacity must follow task sensitivity rather than spread uniformly. Within a layer, a removal cannot be judged by importance alone; its impact also depends on how well the lost contribution can be recovered from what remains. Selection and recovery must therefore be decided jointly. We introduce Carve VFM, which resolves this interdependence. Given a VFM, a calibration set, and a parameter budget, it (i) searches for task-specific capacity allocations across layers; (ii) guides the search via an adaptive recovery-aware fitness criterion; and (iii) reconstructs removed behavior through iterative closed-form low-rank corrections merged into the surviving weights. The result is a dense encoder produced in minutes on a single GPU without backbone retraining. Across twelve benchmarks and four models from DINOv3 and MetaCLIP-2, Carve VFM achieves the highest reported mean accuracy in seven of eight backbone-budget configurations among the evaluated dense structured-pruning methods. Further fine-tuning surpasses the strongest evaluated edge model (88.1% vs. 85.8%), and carved models cut compute and memory while delivering 1.3-2.0x higher GPU throughput than the uncompressed one. These results show that the compact model you need is inside the VFM; it only has to be carved out.
Lotfi Abdelkrim Mecharbat, Abdelrahman Elsayed, Ibrahim Almakky +2
Mohamed bin Zayed University of Artificial Intelligence