Revisiting Model Stitching In the Foundation Model Era

Zheda Mai; Ke Zhang; Fu-En Wang; Zixiao Ken Wang; Albert Y. C. Chen; Lu Xia; Min Sun; Wei-Lun Chao

cs.CVMar 12, 2026

Revisiting Model Stitching In the Foundation Model Era

Authors: Zheda Mai, Ke Zhang, Fu-En Wang, Zixiao Ken Wang, Albert Y. C. Chen, Lu Xia, Min Sun, Wei-Lun Chao,

Organizations: 1The Ohio State University · 3Amazon · 2Boston University

Abstract

Model stitching, connecting early layers of one model (source) to later layers of another (target) via a light stitch layer, has served as a probe of representational compatibility. Prior work finds that models trained on the same dataset remain stitchable (negligible accuracy drop) despite different initializations or objectives. We revisit stitching for Vision Foundation Models (VFMs) that vary in objectives, data, and modality mix (e.g., CLIP, DINOv2, SigLIP 2) and ask: Are heterogeneous VFMs stitchable? We introduce a systematic protocol spanning the stitch points, stitch layer families, training losses, and downstream tasks. Three findings emerge. (1) Stitch layer training matters: conventional approaches that match the intermediate features at the stitch point or optimize the task loss end-to-end struggle to retain accuracy, especially at shallow stitch points. (2) With a simple feature-matching loss at the target model's penultimate layer, heterogeneous VFMs become reliably stitchable across vision tasks. (3) For deep stitch points, the stitched model can surpass either constituent model at only a small inference overhead (for the stitch layer). Building on these findings, we further propose the VFM Stitch Tree (VST), which shares early layers across VFMs while retaining their later layers, yielding a controllable accuracy-latency trade-off for multimodal LLMs that often leverage multiple VFMs. Taken together, our study elevates stitching from a diagnostic probe to a practical recipe for integrating complementary VFM strengths and pinpointing where their representations align or diverge.

Figures & tables

Figure 1: Model stitching training strategies: (a) Layer feature matching trains the stitch layer to match features between the source and target models at the stitch position. (b) Final feature matching trains the stitch layer so that the stitched model matches the target model’s final features. (c) Task loss training optimizes the stitch layer with downstream task objective. Across all strategies, the stitch layer is the only trainable component; the source and target models are kept frozen.

Figure 2: Feature distance of Layer Feature Matching (LFM) and Final Feature Matching (FFM) on SigLIP $\rightarrow$ DINOv2. We measure the $\ell_{2}$ distance between the stitched model’s features and those of the target model when stitching at different positions. indicates the layer feature distance and indicates the final feature distance. While LFM minimizes the layer feature distance, it leaves a relatively large final feature distance. In contrast, FFM achieves substantially smaller final-feature distance.

Figure 3: Final Feature Matching consistently shows better accuracy than Layer Feature Matching. In the DINOv2 $\rightarrow$ SigLIP2 case, the stitched model can even exceed the performance of both constituent models.

Figure 4: Our two-stage training approach (Final Feature Matching + Task Loss Training) allows stitched models to consistently outperform linear-probing of both constituent models.

Pre-train	DINOv2 $\rightarrow$ SigLIP2
Stitch Position	2	6	10	14	18	22
No	25.1	39.4	52.6	62.3	68.6	68.6
LFM	33.1	57.8	59.3	67.3	68.4	70.3
FFM	51.7	59.8	61.1	68.0	72.0	71.8
Pre-train	SigLIP2 $\rightarrow$ DINOv2
Stitch Position	2	6	10	14	18	22

Table 1: Initializing the stitch layer with FFM substantially improves over naive TLT, especially at shallow stitch positions where naive TLT severely underperforms. FFM also provides consistent gains at deeper stitch positions. Compared with LFM initialization, FFM offers a clear advantage at shallow stitch positions, while their performance becomes similar at deeper stitch positions (Numbers are accuracy on fMoW).

Figure 5: Stitched Model vs Self-Stitch. Both DINOv2→SigLIP and SigLIP→DINOv2 (solid lines) consistently outperform their respective self-stitch baselines (dashed lines), demonstrating genuine knowledge fusion gains.

	Classification			Segmentation
	fMoW	iNaturalist	Aircraft	ADE20K
Stitch Position	6 / 14 / 22	6 / 14 / 22	6 / 14 / 22	14 / 22
DINOv2→DINOv2	41.5 / 59.7 / 69.9	56.9 / 81.5 / 91.2	37.8 / 79.3 / 91.2	35.4 / 50.9
SigLIP2→SigLIP2	50.5 / 62.0 / 68.9	71.2 / 88.5 / 87.3	67.9 / 88.1 / 89.3	44.5 / 50.5
DINOv2→SigLIP2	59.8 / 68.0 / 71.8	75.9 / 89.1 / 92.8	77.8 / 87.6 / 92.4	44.9 / 51.2
SigLIP2→DINOv2	53.8 / 69.6 / 72.2	86.3 / 88.9 / 91.9	80.7 / 89.0 / 91.0	49.0 / 51.4

Table 2: Comprehensive results across all datasets and tasks. Classification results in accuracy (%), segmentation in mIoU (%). All results use the two-stage approach (Final Feature Matching + Task Loss Training).

Figure 6: Evaluation of DINOv2, DINOv3, SigLIP2 and CLIP. The stitched models generally outperform both corresponding self-stitch baselines across stitched layers, except for using CLIP as the source model (discussion in Sec. 5.2 ).

Stitch Position		2	6	10	14	18	22
D→S	Linear	26.1	54.3	59.5	66.5	69.1	69.6
	MLP	51.7	55.8	59.3	68.0	72.0	71.8
	LoRA	49.1	49.4	57.4	61.7	67.7	67.3
S→D	Linear	50.3	56.4	60.0	65.7	69.6	71.9
	MLP	53.8	53.8	61.9	69.6	70.4	72.2
	LoRA	48.3	56.2	62.4	65.3	66.2	65.0

Table 3: Stitch layer comparison: MLP consistently outperforms Linear and the LoRA option across stitching directions (D→S, S→D) where D and S denote DINOv2 and SigLIP2, respectively.

	fMoW		iNaturalist
	T-Spec.	T-Agn.	T-Spec.	T-Agn.
D $\rightarrow$ S	54.7	53.8	77.3	77.45
S $\rightarrow$ D	50.8	50.0	75.5	77.7

Table 4: Comparison between task-specific (T-Spec.) and task-agnostic (T-Agn.) stitching. Results are reported for layer 22 (earlier analysis identifies as a strong stitch position).

Figure 7: VFM Stitch Tree (VST) serves as a simple and efficient alternative to naively running all VFMs end-to-end.

Method	Param (M)	GFLOPs	Lat. (ms)	Mem (MB)
Naive	1581.4	878.8	96.2	6075
VST-6	1219.0	701.7	71.4	4502
VST-14	815.9	419.5	43.9	3065
VST-22	412.9	217.4	20.6	1527

Table 5: Efficiency metrics for different stitching configurations. Latency is measured on a single NVIDIA A100 GPU with a batch size of 1. VST significantly reduces computational overhead compared to the Naive Multi-VFM baseline.

Figure 8: VFM Stitch Tree (VST) can be easily applied in various multimodal systems. This is an example of VST in a MoF-LLaVA with CLIP and DINOv2.

Figure 9: VST recovers 45% and 84% of the gain achieved by the naive multi-VFM baseline (100% extra resources), with only 4.3% and 39% extra resources. Details are in Appendix.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Model stitching training strategies: (a) Layer feature matching trains the stitch layer to match features between the source and target models at the stitch position. (b) Final feature matching trains the stitch layer so that the stitched model matches the target model’s final features. (c) Task loss training optimizes the stitch layer with the downstream task objective. (d) Self-Stitch inserts the stitch layer into the source-only and target-only models at the same stitch position. Across all strategies, the stitch layer is the only trainable component; the source and target models are kept frozen.

Layer Feature Matching
	DINOv2			SigLIP2
Linear Probing	46.7			53.5
Layer	2	6	10	14	18	22
DINOv2 $\to$ SigLIP2	49.4	48.2	51.0	48.5	45.3	48.2
SigLIP2 $\to$ DINOv2	29.9	32.4	26.9	42.4	43.7	47.2
Final Feature Matching

Appendix

Table 6: Comparison of Layer and Final Feature Matching. We report linear probing accuracy for different stitching strategies. Orange indicates performance surpassing the SigLIP2 baseline, while Blue indicates performance surpassing the DINOv2 baseline.

	DINOv2		CLIP
Linear Probing	46.7		46.4
Layer	6	14	18	22
DINOv2 $\to$ DINOv2	41.5	59.7	68.2	69.9
CLIP $\to$ CLIP	48.5	63.1	59.6	60.7
CLIP $\to$ DINOv2	53.1	65.8	63.3	64.0
DINOv2 $\to$ CLIP	54.1	68.1	72.0	71.9

Appendix

Table 7: Stitched Model vs Self-Stitch vs Linear Probing: DINOv2 and CLIP.

	DINOv2		SigLIP2
Linear Probing	46.7		53.5
Layer	6	14	18	22
DINOv2 $\to$ DINOv2	41.5	59.7	68.2	69.9
SigLIP2 $\to$ SigLIP2	50.5	62.0	69.4	68.9
SigLIP2 $\to$ DINOv2	53.8	69.6	70.4	72.2
DINOv2 $\to$ SigLIP2	55.8	68.0	72.0	71.8

Appendix

Table 8: Stitched Model vs Self-Stitch vs Linear Probing: DINOv2 and SigLIP2.

	SigLIP2		CLIP
Linear Probing	53.5		46.4
Layer	6	14	18	22
SigLIP2 $\to$ SigLIP2	50.5	62.0	69.4	68.9
CLIP $\to$ CLIP	48.5	63.1	59.6	60.7
CLIP $\to$ SigLIP2	48.3	68.9	65.4	62.9
SigLIP2 $\to$ CLIP	59.8	70.7	73.2	71.9

Appendix

Table 9: Stitched Model vs Self-Stitch vs Linear Probing: SigLIP2 and CLIP.

	DINOv3		SigLIP2
Linear Probing	50.0		53.5
Layer	6	14	18	22
DINOv3 $\to$ DINOv3	44.7	63.9	66.9	69.0
SigLIP2 $\to$ SigLIP2	50.5	62.0	69.4	68.9
SigLIP2 $\to$ DINOv3	58.6	67.0	69.4	69.3
DINOv3 $\to$ SigLIP2	45.0	69.4	70.5	72.4

Appendix

Table 10: Stitched Model vs Self-Stitch vs Linear Probing: DINOv3 and SigLIP2.

	DINOv2		DINOv3
Linear Probing	46.7		50.0
Layer	6	14	18	22
DINOv2 $\to$ DINOv2	41.5	59.7	68.2	69.9
DINOv3 $\to$ DINOv3	44.7	63.9	66.9	69.0
DINOv3 $\to$ DINOv2	43.8	67.2	67.9	72.0
DINOv2 $\to$ DINOv3	57.8	65.0	69.2	70.3

Appendix

Table 11: Stitched Model vs Self-Stitch vs Linear Probing: DINOv2 and DINOv3.

fMoW
Scenario Acc. DINOv2 $\rightarrow$ DINOv2 W R W R R W 69.9 SigLIP2 $\rightarrow$ SigLIP2 R W W R W R 68.9 SigLIP2 $\rightarrow$ DINOv2 R R R W W W 72.2 Count 116 78 47 40 86 33	Scenario Acc. DINOv2 $\rightarrow$ DINOv2 W R W R R W 69.9 SigLIP2 $\rightarrow$ SigLIP2 R W W R W R 68.9 DINOv2 $\rightarrow$ SigLIP2 R R R W W W 71.8 Count 77 116 54 52 48 72
iNaturalist
Scenario Acc. DINOv2 $\rightarrow$ DINOv2 W R W R R W 91.2 SigLIP2 $\rightarrow$ SigLIP2 R W W R W R 87.3 SigLIP2 $\rightarrow$ DINOv2 R R R W W W 91.9 Count 138 292 75 68 116 53	Scenario Acc. DINOv2 $\rightarrow$ DINOv2 W R W R R W 91.2 SigLIP2 $\rightarrow$ SigLIP2 R W W R W R 87.3 DINOv2 $\rightarrow$ SigLIP2 R R R W W W 92.8 Count 127 331 80 49 77 64

Appendix

Table 12: Analysis of Predictions: R and W denote Right and Wrong predictions for each model, with counts in the last row and accuracies in the last column for fMoW and iNaturalist. Stitched models fall into complementarity scenarios (stitched is correct while at least one self-stitched baseline is wrong) far more often than interference ones (stitched is wrong while at least one self-stitched baseline is correct). When the two self-stitched models disagree, the stitched model tends to follow the source model on fMoW but follows the stronger model DINOv2 on iNaturalist.

Number of VFMs	Full	VST-6	VST-14	VST-22
2	100%	74%	39%	4%
3	200%	148%	78%	9%
4	300%	222%	117%	13%
5	400%	296%	156%	17%

Appendix

Table 13: Comparison of Additional Computation Cost. We report the additional increase in computation compared to a single VFM. “Full” denotes running all VFMs independently. “VST-

n

” denotes a shared backbone up to the stitch position

n

, where layers

1

through

n

are shared and subsequent layers are executed independently. For example, VST-14 shares the first 14 layers and maintains specialized branches from layer 15 onwards.

1. Define the Upper Bound (Denominator)
Model Configuration	VQAv2			MME		Efficiency Metrics
	Yes/No	Number	Other	Percep.	Cogn.	Avg Gain %	Add. Cost
(A) CLIP Baseline	91.75	58.74	69.00	1418.5	277.1	-	0%
(B) Full (CLIP + DINOv2)	92.72	61.64	70.30	1460.3	311.8	-	100%
(C) Max Gain ( $\Delta_{\text{max}}=B-A$ )	0.97	2.90	1.30	41.8	34.7	(Denom.)	-
2. VST-22: The Lightweight Knob (High Efficiency)

Appendix

Table 14: Detailed Calculation of Normalized Gain. Comparison of the “Lightweight” VST-22 and the “Balanced” VST-14. The Orange Row represents the gain (

\Delta_{\text{max}}

) achieved by the computationally expensive Naive ensemble (100% Cost). The Pink Rows represent the actual gain (

\Delta_{\text{VST}}

) achieved by our method. The Normalized Gain is calculated as

\Delta_{\text{VST}}/\Delta_{\text{max}}

. Note that VST-22 achieves nearly half the total gain (45.5%) with negligible cost (4.3%).

Vision Foundation Models

Vision foundation models are large pretrained visual systems adapted to recognition, segmentation, geometry, and video tasks.

See all 139 papers

Stitched Value Model for Diffusion Alignment
May 19, 2026Hyojun Go, Hyungjin Chung, Prune Truong +8Diffusion Model Alignment Diffusion Model Guidance
1ETH Zurich · 2Google · University of Copenhagen
KLAS: Using Similarity to Stitch Neural Networks for Improved Accuracy-Efficiency Tradeoffs
May 28, 2026Debopam Sanyal, Anantharaman Iyer, Alind Khare +5Efficient Neural Network Inference Representational Similarity Analysis
Georgia Institute of Technology · Microsoft M365 Research · Cisco Research +1
Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models
Jun 3, 2026Shuwen Yu, Zhanxuan Hu, Yi Zhao +2Cross-Modal Representation Learning Vision Foundation Model Adaptation
Yunnan Normal University, Kunming, China · Kunming University of Science and Technology, Kunming, China

May 19, 2026J/K move · Enter open · S save

Stitched Value Model for Diffusion Alignment

Hyojun Go, Hyungjin Chung, Prune Truong, Goutam Bhat+7

1ETH Zurich · 2Google · University of Copenhagen

Diffusion Model Alignment Diffusion Model Guidance RL for Diffusion Models

Open paper AskPDF arXiv

For practical use, diffusion- or flow-based generative models must be aligned with task-specific rewards, such as prompt fidelity or aesthetic preference. That alignment is challenging because the reward is defined for clean output images, but the alignment procedure requires value function estimates at noisy intermediate latents. Existing methods resort to Tweedie-style or Monte Carlo approximations, trading off estimator bias against computational cost: Tweedie estimates are efficient but biased, while Monte Carlo estimates are more accurate but require expensive rollouts. A natural alternative would be a learned value function, but it remains an open question how to effectively train a strong and general value model specifically for noisy latents. Here, we propose StitchVM, a model stitching framework that efficiently transfers reward models pretrained for clean images to the noisy latent regime. StitchVM starts from an existing, truncated pixel-space reward model and attaches a frozen diffusion backbone to it as its head. From the pixel-space model, the resulting hybrid retains a carefully pretrained, robust reward capability; from the diffusion backbone, it inherits its native ability to handle noisy latents. The stitching procedure is exceptionally lightweight, e.g., stitching and finetuning CLIP ViT-L and SD 3.5 Medium takes only 10 GPU-hours. By lifting powerful pixel-space reward models to latent space, StitchVM opens up a new style of diffusion alignment: instead of rough, yet costly per-sample approximation of the value function, the correct function for the actual, noisy latents is constructed once and then amortized over many samples and iterations. We show that this approach yields improvements across a broad range of downstream steering and post-training methods: DPS becomes

3.2\times

faster while halving peak GPU memory, and DiffusionNFT becomes

2.3\times

faster.

Abstract

Vision Foundation Models

Explore similar work

Stitched Value Model for Diffusion Alignment

KLAS: Using Similarity to Stitch Neural Networks for Improved Accuracy-Efficiency Tradeoffs

Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models