Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models

Jiawei Fan; Shigeng Wang; Chao Li; Xiaolong Liu; Anbang Yao

cs.CVApr 14, 2026

Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models

Authors: Jiawei Fan, Shigeng Wang, Chao Li, Xiaolong Liu, Anbang Yao

Organizations: Intel Labs China · iMotion Automotive Technology

Abstract

In this paper, we present Chain-of-Models Pre-Training (CoM-PT), a novel performance-lossless training acceleration method for vision foundation models (VFMs). This approach fundamentally differs from existing acceleration methods in its core motivation: rather than optimizing each model individually, CoM-PT is designed to accelerate the training pipeline at the model family level, scaling efficiently as the model family expands. Specifically, CoM-PT establishes a pre-training sequence for the model family, arranged in ascending order of model size, called model chain. In this chain, only the smallest model undergoes standard individual pre-training, while the other models are efficiently trained through sequential inverse knowledge transfer from their smaller predecessors by jointly reusing the knowledge in the parameter space and the feature space. As a result, CoM-PT enables all models to achieve performance that is mostly superior to standard individual training while significantly reducing training cost, and this is extensively validated across 45 datasets spanning zero-shot and fine-tuning tasks. Notably, its efficient scaling property yields a remarkable phenomenon: training more models even results in higher efficiency. For instance, when pre-training on CC3M: i) given ViT-L as the largest model, progressively prepending smaller models to the model chain reduces computational complexity by up to 72%; ii) within a fixed model size range, as the VFM family scales across 3, 4, and 7 models, the acceleration ratio of CoM-PT exhibits a striking leap: from 4.13X to 5.68X and 7.09X. Since CoM-PT is naturally agnostic to specific pre-training paradigms, we open-source the code to spur further extensions in more computationally intensive scenarios, such as large language model pre-training.

Figures & tables

Figure 1: Standard individual pre-training vs . Chain-of-Models Pre-Training . All ViTs [ 17 ] are trained on a combined dataset of CC3M [ 59 ] and CC12M [ 4 ] , and evaluated on ImageNet-1K [ 14 ] .

Figure 2: Overview of Chain-of-Models Pre-training pipeline with inverse knowledge transfer relay. Models are organized sequentially by ascending model size, from the smallest model

m_{1}

to the largest

m_{n}

, forming a model chain.

m_{1}

first undergoes standard individual pre-training. Each subsequent

m_{i}

is then efficiently trained via inverse knowledge transfer (denoted as

\mathrel{\hbox to9.9pt{\vbox to7.2pt{\pgfpicture\makeatletter\hbox{\hskip 4.23409pt\lower-3.59999pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {{{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#FFFFFF} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.9pt} \lx@inpgf@ignorespaces{{}{}{}{}{}{}{}{}{{}{}}{}{{}{}\lx@inpgf@ignorespaces}{{}{}\lx@inpgf@ignorespaces}{{{}{}}}{{}{}\lx@inpgf@ignorespaces}{{{}{}}}{{}{}}{{}{}\lx@inpgf@ignorespaces}{{}{}\lx@inpgf@ignorespaces}{{}{}\lx@inpgf@ignorespaces}{}{}{}{{}{}\lx@inpgf@ignorespaces}{{}{}\lx@inpgf@ignorespaces}{{}{}\lx@inpgf@ignorespaces}{}{}{}{{}{}\lx@inpgf@ignorespaces}{{}{}\lx@inpgf@ignorespaces}{{}{}\lx@inpgf@ignorespaces}{}{}{}{}{}{}{}{}{}{}{{}}{}{{{}}{{\lx@inpgf@ignorespaces}}{}{}{}{}{}{}{}}{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#FFFFFF} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.9pt} \lx@inpgf@ignorespaces{}\lxSVG@fillstroke\lxSVG@drawpath@unclipped{M 7.22 0 L 2.86 4.36 L 2.86 1.98 L -5.24 1.98 L -5.24 -1.98 L 2.86 -1.98 L 2.86 -4.36 Z}{} \lx@inpgf@ignorespaces \lxSVG@closescope }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{58}\lxSVG@closescope }}} \lxSVG@closescope }}} } \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}

) from its immediate predecessor

m_{i-1}

, forming a continuous relay along the model chain. Details of our proposed inverse knowledge transfer are in Figure 3 .

Figure 3: Inverse knowledge transfer from

m_{i}

m_{i+1}

Figure 4: Convergence analysis across different training epochs. We train ViT-T/16, ViT-S/16, and ViT-B/16 on CC3M for 32, 64, 128, and 256 epochs to determine convergence points. Top-1 (%) represents zero-shot classification accuracy on ImageNet-1K.

Method	Text Augmentation Sources	# Epochs	Top-1(%)
CLIP	-	32	16.00
LaCLIP	LLaMA [ 64 ]	32	21.50
MLLM-A	Multi-MLLMs [ 83 , 32 , 2 , 40 ]	32	25.00
Our Baseline	ShareGPT4V [ 7 ]	32	28.80
(LaCLIP)	ShareGPT4V [ 7 ]	128	31.80

Table 1: Performance comparison with recent top methods on the CC3M dataset. Top-1 (%) represents zero-shot classification accuracy on ImageNet-1K. Our method achieves similar acceleration with baseline using 32 epochs, demonstrated in Figure 9(a) .

Model Family	Method	Model	Zero-shot Top-1/R@1 (%)			Training MACs ( $10^{10}$ G)		Training Time ( $10^{2}$ GPU hrs)
			ImageNet-1K	VTAB+	COCO	Individual	Accumulated	Individual	Accumulated
Pre-training Dataset: CC3M
ViT Family			Baseline	ViT-T/16	26.16	25.61	29.06	0.62	0.62	1.54	1.54
	ViT-S/16	30.16		28.84	32.81	1.54	2.16	2.40	3.94
	ViT-B/16	31.80		30.08	34.93	3.75	5.91	3.45	7.39
	ViT-L/16	33.77		30.39	36.53	10.79	16.70	7.56	14.95

Table 2: Main results. Evaluation is conducted on three benchmarks: ImageNet-1K, VTAB+, and COCO. Efficiency is measured in terms of training MACs and training time. Specifically, training MACs include all computations in the forward and backward propagations, as well as parameter updates, and training time denotes the required NVIDIA A100 (80GB) GPU hours for pre-training. ‘Individual’ and ‘accumulated’ refer to specific model overhead and cumulative overhead along the model chain, respectively. Best results are bolded .

Figure 5: Comparison of acceleration ratios, measured by two efficiency metrics. We report both training MACs and A100 GPU hours for the ViT family pre-trained on the CC3M dataset.

		Open-Vocabulary Semantic Segmentation Tasks						Vision-Language Tasks
Backbone	PT Method	ADE-847	ADE-150	PC-459	PC-59	VOC	Avg.	TextVQA	ScienceQA	POPE	VQAv2	Avg.
ViT-B/16	Baseline	0.45	11.40	1.07	36.12	68.59	23.53	43.08	62.57	73.62	54.16	58.36
ViT-B/16	CoM-PT	3.18	18.30	6.68	40.73	76.96	29.17	43.21	63.41	74.17	54.59	58.85
ViT-L/16	Baseline	1.03	13.58	4.47	33.99	69.76	24.57	43.64	63.56	74.50	55.68	59.35
ViT-L/16	CoM-PT	4.73	20.14	7.00	42.17	79.52	30.71	43.83	63.51	75.15	56.07	59.64

Table 3: Fine-tuning on downstream tasks. With backbones pre-trained by baseline method and CoM-PT, the framework SAN [ 70 ] for open-vocabulary semantic segmentation is fine-tuned on the COCO-Stuff [ 3 ] training set, while the LLaVA-1.5-7B [ 40 ] for vision-language tasks undergoes LoRA-based visual instruction fine-tuning following its official protocol. Both ViT-B/16 and ViT-L/16 are pre-trained on the CC3M dataset. The ‘PT method’ refers to the pre-training method of the backbone. Best results are bolded .

Figure 6: Reduced training MACs vs. model chain starting with smaller models. The reduced MACs represent the total training cost of the model chain relative to the cost of individually pre-training ViT-L. All models are performance-lossless.

Figure 7: Training MACs and acceleration ratios of model chains with different expansion ratios . Experiments are conducted on the CC3M dataset. All models are performance-lossless.

Figure 8: Training epoch allocation. Along model chain structured ViT-T $\rightarrow$ ViT-S $\rightarrow$ ViT-B $\rightarrow$ ViT-L, we determine the minimum training epochs required per model to achieve lossless acceleration via CoM-PT, compared to standard 128-epoch baselines on CC3M.

Table 4: Ablation Studies. (a) We study efficiency gain from the model chain, by comparing models progressively trained along the model chain (i.e., ViT-T

\rightarrow

ViT-S

\rightarrow

ViT-B) against those independently initialized from a trained ViT-T. (b) We study the complementarity of inverse weight initialization (IWI) and inverse feature distillation (IFD) using ViT-T

\rightarrow

ViT-S short chain. All models are trained on CC3M.

Panel of Figure 9 — Figure 9: Acceleration ratio variation under different training epochs and data scales. Experiments use the ViT family. (a) On CC3M, we vary the training epochs (32, 64, and 128). (b) We test the data scale from 22.0M to 206.8M under a fixed 64-epoch setting to isolate data scale effects. The curve fitting format follows [ 77 ] .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	# Total Pairs	# Collected Pairs	# Augmented Pairs
CC3M	3,318,333	2,753,427	44,054,832
CC12M	12,423,374	10,173,631	162,778,096
Merged-15M	-	12,927,058	206,832,928

Appendix

Table A : CC3M, CC12M and our Merged-15M datasets used for pre-training in this paper. Note that we only report the number of samples in the training set.

Dataset	Abbr.	Test size
ImageNet-1K	-	50,000
ImageNet-v2	-	10,000
ImageNet-R	-	30,000
ImageNet Sketch	-	50,889
ObjectNet	-	18,574
ImageNet-A	-	7,500

Appendix

Table B : Datasets in VTAB+ with abbreviations and test sizes.

Table C : Detailed settings of baseline pre-training and chain-of-models pre-training on CC3M and Merged-15M datasets.

Dataset	ViT Family					Swin Family
Dataset	ViT-T/16	ViT-S/16	ViT-M/16	ViT-B/16	ViT-L/16	Swin-T	Swin-S	Swin-B	Swin-L
CC3M	128	24	–	18	15	128	24	16	12
Merged-15M	–	64	21	17	15	–	–	–	–

Appendix

Table D: Training epochs for different models in main experiments.

Table E : Detailed settings of fine-tuning on COCO-Stuff for open-vocabulary semantic segmentation tasks.

Table F : Detailed settings of feature alignment training and LoRA-based visual instruction fine-tuning.

Model Family	Model	Image Encoder			Text Encoder			Overall
		Width	Depth	Param (M)	Width	Depth	Param (M)	Param (M)	Forward MACs (G)	Training MACs (G)
ViT Family	ViT-T/16	192	12	5.62	256	12	9.48	15.10	5.82	17.57
	ViT-C/16	256	12	9.86	320	12	14.80	24.66	9.23	27.87
	ViT-S/16	384	12	21.81	384	12	21.29	43.10	14.44	43.65
	ViT-M/16	512	12	38.59	448	12	28.97	67.56	21.88	66.17
	ViT-B/16	768	12	86.19	512	12	37.83	124.02	35.09	106.27

Appendix

Table G : Architecture specifications for VFM families. Forward MACs and Training MACs represent the computational complexity of the model for a sample (an image and its corresponding 4 text descriptions) in our experiment.

Figure A : Detailed illustrations of weight initialization in width and depth. First, (a)-(c) illustrates different weight initialization approaches for width differences. Duplication (a) involves copying the teacher’s weights multiple times and directly assigning these copies to the student. Interpolation (b) conducts bilinear operations to expand the teacher’s weights to fit the shape of the student. Insertion (c) involves directly assigning the teacher’s weights to the student, with all remaining weights initialized randomly. Second, (d)-(f) show the different weight initialization methods for depth differences. (d) only initializes the first

p

layers across the total

q

layers, while the interval insertion (e) skips several layers by calculating

q/p

. As a combination, the duplicate insertion (f) initializes the skipped layers by duplicating the weights of the preceding layer.

Figure B : Ablation study on various simple weight initialization approaches. For width initialization, experiments are conducted on the ViT-T/16 $\rightarrow$ ViT-S/16 sub-chain. Progressively, with the best simple width initialization method, we explore depth initialization on the ViT-B/16 $\rightarrow$ ViT-L/16 sub-chain. Inverse feature distillation is applied as default. Details for each method are in Figure A .

Method	ZS Performance
Method	ImageNet-1K	COCO
Baseline	26.48	28.34
+IFD	29.29	32.10
+IFD+IWI	30.24	34.15
+IFD+Net2Net [ 9 ]	30.07	33.63
+IFD+NetExpand [ 16 ]	30.36	34.13

Appendix

Table H : Our simple approaches vs. previous sophisticated designs. Experiments are conducted using the ViT-T/16

\rightarrow

ViT-S/16 sub-chain, trained on the CC3M dataset.

Figure C : Validation curves of different choices in Table 4(b) of the main paper.

Panel of Figure 25 — Figure D : Ablation study on inverse feature distillation magnitude. For better illustration, we define $r$ as the ratio between the magnitude of $L_{IFD}$ and $L_{task}$ . The experiment is conducted on ViT-T/16 $\rightarrow$ ViT-S/16 on the CC3M dataset. We show how IFD enables ViT-S/16 to achieve target performance (128-epoch individual pre-training) with only 24 epochs. The teacher model is ViT-T/16 trained individually for 128 epochs. Inverse weight initialization is applied as default.

PT Dataset	CC3M														Merged-15M
Models	ViT-T/16	ViT-S/16		ViT-B/16		ViT-L/16		Swin-T	Swin-S		Swin-B		Swin-L		ViT-S/16	ViT-M/16		ViT-B/16		ViT-L/16
PT Method	baseline	baseline	CoM-PT	baseline	CoM-PT	baseline	CoM-PT	baseline	baseline	CoM-PT	baseline	CoM-PT	baseline	CoM-PT	baseline	baseline	CoM-PT	baseline	CoM-PT	baseline	CoM-PT
ImageNet-1K	26.16	30.16	30.24	31.80	31.83	33.77	34.27	33.84	35.88	36.19	36.13	36.71	36.77	37.06	43.97	45.90	45.85	47.15	47.39	48.80	49.47
ImageNet-v2	22.85	26.47	25.91	27.42	28.00	29.39	29.94	29.00	30.99	31.31	31.54	31.59	31.83	32.30	38.24	39.52	39.85	41.35	40.79	43.06	42.84
ImageNet-R	37.15	42.36	40.77	44.09	43.76	47.60	47.94	47.35	50.70	50.72	52.16	52.04	51.85	52.96	63.28	66.06	65.61	69.16	67.70	71.65	71.02
ImageNet Sketch	16.93	21.24	20.53	22.81	21.32	24.54	24.76	24.92	26.25	26.80	27.66	27.12	27.42	27.52	35.09	36.74	36.96	39.27	38.69	41.06	41.10

Appendix

Table I : Zero-shot Top-1 accuracy (%) across the 35 datasets of the VTAB+ benchmark. Detailed results corresponding to the VTAB+ averages presented in Table 2 of the main paper.

Table 27

Vision Foundation Models

Vision foundation models are large pretrained visual systems adapted to recognition, segmentation, geometry, and video tasks.

See all 139 papers

Accelerating Vision Foundation Models with Drop-in Depthwise Convolution
May 21, 2026Carmelo Scribano, Mohammad Mahdi, Nedyalko Prisadnikov +5Vision Foundation Models Vision Foundation Model Adaptation
University of Modena and Reggio Emilia, Modena, Italy. · INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria
A note on data-efficient finetuning of vision foundation models
Oct 27, 2026Niclas Hergenröther, Antonio OrvietoVision Foundation Model Adaptation Vision Foundation Models
ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center
Carve VFM: Carving Specialists out of Generalists, Automatic Task-Specific Compression of Vision Foundation Models
Oct 12, 2026Lotfi Abdelkrim Mecharbat, Abdelrahman Elsayed, Ibrahim Almakky +2Model Compression Neural Network Pruning
Mohamed bin Zayed University of Artificial Intelligence

May 21, 2026J/K move · Enter open · S save

Accelerating Vision Foundation Models with Drop-in Depthwise Convolution

Carmelo Scribano, Mohammad Mahdi, Nedyalko Prisadnikov, Yuqian Fu+4

University of Modena and Reggio Emilia, Modena, Italy. · INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria

Vision Foundation Models Vision Foundation Model Adaptation Efficient ViTs Convolutional Neural Networks Depthwise Separable Convolution

Open paper AskPDF arXiv

Pretrained vision foundation models deliver strong performance across tasks with limited fine-tuning. However, their Vision Transformer (ViT) backbones impose high inference costs, limiting deployment on resource-constrained devices. In this work, we accelerate large-scale pretrained ViTs while preserving their feature extraction capabilities by exploiting the intrinsic convolution-like behavior of some attention heads. Specifically, we introduce an efficient depthwise convolution-based layer that serves as a drop-in replacement for these heads. Additionally, we propose simple strategies to identify which heads can be replaced and introduce a fine-tuning procedure that recovers downstream task performance. Across both image classification and segmentation tasks, our method achieves 17-20% percent inference speedup with minimal performance degradation. We validate the approach through detailed derivations, extensive experiments, and efficiency benchmarks. The reference implementation is publicly available.

Abstract

Vision Foundation Models

Explore similar work

Accelerating Vision Foundation Models with Drop-in Depthwise Convolution

A note on data-efficient finetuning of vision foundation models

Carve VFM: Carving Specialists out of Generalists, Automatic Task-Specific Compression of Vision Foundation Models