Model stitching, connecting early layers of one model (source) to later layers of another (target) via a light stitch layer, has served as a probe of representational compatibility. Prior work finds that models trained on the same dataset remain stitchable (negligible accuracy drop) despite different initializations or objectives. We revisit stitching for Vision Foundation Models (VFMs) that vary in objectives, data, and modality mix (e.g., CLIP, DINOv2, SigLIP 2) and ask: Are heterogeneous VFMs stitchable? We introduce a systematic protocol spanning the stitch points, stitch layer families, training losses, and downstream tasks. Three findings emerge. (1) Stitch layer training matters: conventional approaches that match the intermediate features at the stitch point or optimize the task loss end-to-end struggle to retain accuracy, especially at shallow stitch points. (2) With a simple feature-matching loss at the target model's penultimate layer, heterogeneous VFMs become reliably stitchable across vision tasks. (3) For deep stitch points, the stitched model can surpass either constituent model at only a small inference overhead (for the stitch layer). Building on these findings, we further propose the VFM Stitch Tree (VST), which shares early layers across VFMs while retaining their later layers, yielding a controllable accuracy-latency trade-off for multimodal LLMs that often leverage multiple VFMs. Taken together, our study elevates stitching from a diagnostic probe to a practical recipe for integrating complementary VFM strengths and pinpointing where their representations align or diverge.
Figures & tables
Figure 1: Model stitching training strategies: (a) Layer feature matching trains the stitch layer to match features between the source and target models at the stitch position. (b) Final feature matching trains the stitch layer so that the stitched model matches the target model’s final features. (c) Task loss training optimizes the stitch layer with downstream task objective. Across all strategies, the stitch layer is the only trainable component; the source and target models are kept frozen.
Figure 2: Feature distance of Layer Feature Matching (LFM) and Final Feature Matching (FFM) on SigLIP → DINOv2. We measure the ℓ2 distance between the stitched model’s features and those of the target model when stitching at different positions. indicates the layer feature distance and indicates the final feature distance. While LFM minimizes the layer feature distance, it leaves a relatively large final feature distance. In contrast, FFM achieves substantially smaller final-feature distance.
Figure 3: Final Feature Matching consistently shows better accuracy than Layer Feature Matching. In the DINOv2 → SigLIP2 case, the stitched model can even exceed the performance of both constituent models.
Figure 4: Our two-stage training approach (Final Feature Matching + Task Loss Training) allows stitched models to consistently outperform linear-probing of both constituent models.
Pre-train
DINOv2 → SigLIP2
Stitch Position
2
6
10
14
18
22
No
25.1
39.4
52.6
62.3
68.6
68.6
LFM
33.1
57.8
59.3
67.3
68.4
70.3
FFM
51.7
59.8
61.1
68.0
72.0
71.8
Pre-train
SigLIP2 → DINOv2
Stitch Position
2
6
10
14
18
22
Table 1: Initializing the stitch layer with FFM substantially improves over naive TLT, especially at shallow stitch positions where naive TLT severely underperforms. FFM also provides consistent gains at deeper stitch positions. Compared with LFM initialization, FFM offers a clear advantage at shallow stitch positions, while their performance becomes similar at deeper stitch positions (Numbers are accuracy on fMoW).
Figure 5: Stitched Model vs Self-Stitch. Both DINOv2→SigLIP and SigLIP→DINOv2 (solid lines) consistently outperform their respective self-stitch baselines (dashed lines), demonstrating genuine knowledge fusion gains.
Classification
Segmentation
fMoW
iNaturalist
Aircraft
ADE20K
Stitch Position
6 / 14 / 22
6 / 14 / 22
6 / 14 / 22
14 / 22
DINOv2→DINOv2
41.5 / 59.7 / 69.9
56.9 / 81.5 / 91.2
37.8 / 79.3 / 91.2
35.4 / 50.9
SigLIP2→SigLIP2
50.5 / 62.0 / 68.9
71.2 / 88.5 / 87.3
67.9 / 88.1 / 89.3
44.5 / 50.5
DINOv2→SigLIP2
59.8 / 68.0 / 71.8
75.9 / 89.1 / 92.8
77.8 / 87.6 / 92.4
44.9 / 51.2
SigLIP2→DINOv2
53.8 / 69.6 / 72.2
86.3 / 88.9 / 91.9
80.7 / 89.0 / 91.0
49.0 / 51.4
Table 2: Comprehensive results across all datasets and tasks. Classification results in accuracy (%), segmentation in mIoU (%). All results use the two-stage approach (Final Feature Matching + Task Loss Training).
Figure 6: Evaluation of DINOv2, DINOv3, SigLIP2 and CLIP. The stitched models generally outperform both corresponding self-stitch baselines across stitched layers, except for using CLIP as the source model (discussion in Sec. 5.2 ).
Stitch Position
2
6
10
14
18
22
D→S
Linear
26.1
54.3
59.5
66.5
69.1
69.6
MLP
51.7
55.8
59.3
68.0
72.0
71.8
LoRA
49.1
49.4
57.4
61.7
67.7
67.3
S→D
Linear
50.3
56.4
60.0
65.7
69.6
71.9
MLP
53.8
53.8
61.9
69.6
70.4
72.2
LoRA
48.3
56.2
62.4
65.3
66.2
65.0
Table 3: Stitch layer comparison: MLP consistently outperforms Linear and the LoRA option across stitching directions (D→S, S→D) where D and S denote DINOv2 and SigLIP2, respectively.
fMoW
iNaturalist
T-Spec.
T-Agn.
T-Spec.
T-Agn.
D → S
54.7
53.8
77.3
77.45
S → D
50.8
50.0
75.5
77.7
Table 4: Comparison between task-specific (T-Spec.) and task-agnostic (T-Agn.) stitching. Results are reported for layer 22 (earlier analysis identifies as a strong stitch position).
Figure 7: VFM Stitch Tree (VST) serves as a simple and efficient alternative to naively running all VFMs end-to-end.
Method
Param (M)
GFLOPs
Lat. (ms)
Mem (MB)
Naive
1581.4
878.8
96.2
6075
VST-6
1219.0
701.7
71.4
4502
VST-14
815.9
419.5
43.9
3065
VST-22
412.9
217.4
20.6
1527
Table 5: Efficiency metrics for different stitching configurations. Latency is measured on a single NVIDIA A100 GPU with a batch size of 1. VST significantly reduces computational overhead compared to the Naive Multi-VFM baseline.
Figure 8: VFM Stitch Tree (VST) can be easily applied in various multimodal systems. This is an example of VST in a MoF-LLaVA with CLIP and DINOv2.
Figure 9: VST recovers 45% and 84% of the gain achieved by the naive multi-VFM baseline (100% extra resources), with only 4.3% and 39% extra resources. Details are in Appendix.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Model stitching training strategies: (a) Layer feature matching trains the stitch layer to match features between the source and target models at the stitch position. (b) Final feature matching trains the stitch layer so that the stitched model matches the target model’s final features. (c) Task loss training optimizes the stitch layer with the downstream task objective. (d) Self-Stitch inserts the stitch layer into the source-only and target-only models at the same stitch position. Across all strategies, the stitch layer is the only trainable component; the source and target models are kept frozen.
DINOv2
SigLIP2
Linear Probing
46.7
53.5
Layer
2
6
10
14
18
22
Layer Feature Matching
DINOv2 → SigLIP2
49.4
48.2
51.0
48.5
45.3
48.2
SigLIP2 → DINOv2
29.9
32.4
26.9
42.4
43.7
47.2
Final Feature Matching
Appendix
Table 6: Comparison of Layer and Final Feature Matching. We report linear probing accuracy for different stitching strategies. Orange indicates performance surpassing the SigLIP2 baseline, while Blue indicates performance surpassing the DINOv2 baseline.
DINOv2
CLIP
Linear Probing
46.7
46.4
Layer
6
14
18
22
DINOv2 → DINOv2
41.5
59.7
68.2
69.9
CLIP → CLIP
48.5
63.1
59.6
60.7
CLIP → DINOv2
53.1
65.8
63.3
64.0
DINOv2 → CLIP
54.1
68.1
72.0
71.9
Appendix
Table 7: Stitched Model vs Self-Stitch vs Linear Probing: DINOv2 and CLIP.
DINOv2
SigLIP2
Linear Probing
46.7
53.5
Layer
6
14
18
22
DINOv2 → DINOv2
41.5
59.7
68.2
69.9
SigLIP2 → SigLIP2
50.5
62.0
69.4
68.9
SigLIP2 → DINOv2
53.8
69.6
70.4
72.2
DINOv2 → SigLIP2
55.8
68.0
72.0
71.8
Appendix
Table 8: Stitched Model vs Self-Stitch vs Linear Probing: DINOv2 and SigLIP2.
SigLIP2
CLIP
Linear Probing
53.5
46.4
Layer
6
14
18
22
SigLIP2 → SigLIP2
50.5
62.0
69.4
68.9
CLIP → CLIP
48.5
63.1
59.6
60.7
CLIP → SigLIP2
48.3
68.9
65.4
62.9
SigLIP2 → CLIP
59.8
70.7
73.2
71.9
Appendix
Table 9: Stitched Model vs Self-Stitch vs Linear Probing: SigLIP2 and CLIP.
DINOv3
SigLIP2
Linear Probing
50.0
53.5
Layer
6
14
18
22
DINOv3 → DINOv3
44.7
63.9
66.9
69.0
SigLIP2 → SigLIP2
50.5
62.0
69.4
68.9
SigLIP2 → DINOv3
58.6
67.0
69.4
69.3
DINOv3 → SigLIP2
45.0
69.4
70.5
72.4
Appendix
Table 10: Stitched Model vs Self-Stitch vs Linear Probing: DINOv3 and SigLIP2.
DINOv2
DINOv3
Linear Probing
46.7
50.0
Layer
6
14
18
22
DINOv2 → DINOv2
41.5
59.7
68.2
69.9
DINOv3 → DINOv3
44.7
63.9
66.9
69.0
DINOv3 → DINOv2
43.8
67.2
67.9
72.0
DINOv2 → DINOv3
57.8
65.0
69.2
70.3
Appendix
Table 11: Stitched Model vs Self-Stitch vs Linear Probing: DINOv2 and DINOv3.
fMoW
Scenario Acc. DINOv2 → DINOv2 W R W R R W 69.9 SigLIP2 → SigLIP2 R W W R W R 68.9 SigLIP2 → DINOv2 R R R W W W 72.2 Count 116 78 47 40 86 33
Scenario Acc. DINOv2 → DINOv2 W R W R R W 69.9 SigLIP2 → SigLIP2 R W W R W R 68.9 DINOv2 → SigLIP2 R R R W W W 71.8 Count 77 116 54 52 48 72
iNaturalist
Scenario Acc. DINOv2 → DINOv2 W R W R R W 91.2 SigLIP2 → SigLIP2 R W W R W R 87.3 SigLIP2 → DINOv2 R R R W W W 91.9 Count 138 292 75 68 116 53
Scenario Acc. DINOv2 → DINOv2 W R W R R W 91.2 SigLIP2 → SigLIP2 R W W R W R 87.3 DINOv2 → SigLIP2 R R R W W W 92.8 Count 127 331 80 49 77 64
Appendix
Table 12: Analysis of Predictions: R and W denote Right and Wrong predictions for each model, with counts in the last row and accuracies in the last column for fMoW and iNaturalist. Stitched models fall into complementarity scenarios (stitched is correct while at least one self-stitched baseline is wrong) far more often than interference ones (stitched is wrong while at least one self-stitched baseline is correct). When the two self-stitched models disagree, the stitched model tends to follow the source model on fMoW but follows the stronger model DINOv2 on iNaturalist.
Number of VFMs
Full
VST-6
VST-14
VST-22
2
100%
74%
39%
4%
3
200%
148%
78%
9%
4
300%
222%
117%
13%
5
400%
296%
156%
17%
Appendix
Table 13: Comparison of Additional Computation Cost. We report the additional increase in computation compared to a single VFM. “Full” denotes running all VFMs independently. “VST- n ” denotes a shared backbone up to the stitch position n , where layers 1 through n are shared and subsequent layers are executed independently. For example, VST-14 shares the first 14 layers and maintains specialized branches from layer 15 onwards.
Model Configuration
VQAv2
MME
Efficiency Metrics
Yes/No
Number
Other
Percep.
Cogn.
Avg Gain %
Add. Cost
1. Define the Upper Bound (Denominator)
(A) CLIP Baseline
91.75
58.74
69.00
1418.5
277.1
-
0%
(B) Full (CLIP + DINOv2)
92.72
61.64
70.30
1460.3
311.8
-
100%
(C) Max Gain ( Δmax=B−A )
0.97
2.90
1.30
41.8
34.7
(Denom.)
-
2. VST-22: The Lightweight Knob (High Efficiency)
Appendix
Table 14: Detailed Calculation of Normalized Gain. Comparison of the “Lightweight” VST-22 and the “Balanced” VST-14. The Orange Row represents the gain ( Δmax ) achieved by the computationally expensive Naive ensemble (100% Cost). The Pink Rows represent the actual gain ( ΔVST ) achieved by our method. The Normalized Gain is calculated as ΔVST/Δmax . Note that VST-22 achieves nearly half the total gain (45.5%) with negligible cost (4.3%).
For practical use, diffusion- or flow-based generative models must be aligned with task-specific rewards, such as prompt fidelity or aesthetic preference. That alignment is challenging because the reward is defined for clean output images, but the alignment procedure requires value function estimates at noisy intermediate latents. Existing methods resort to Tweedie-style or Monte Carlo approximations, trading off estimator bias against computational cost: Tweedie estimates are efficient but biased, while Monte Carlo estimates are more accurate but require expensive rollouts. A natural alternative would be a learned value function, but it remains an open question how to effectively train a strong and general value model specifically for noisy latents. Here, we propose StitchVM, a model stitching framework that efficiently transfers reward models pretrained for clean images to the noisy latent regime. StitchVM starts from an existing, truncated pixel-space reward model and attaches a frozen diffusion backbone to it as its head. From the pixel-space model, the resulting hybrid retains a carefully pretrained, robust reward capability; from the diffusion backbone, it inherits its native ability to handle noisy latents. The stitching procedure is exceptionally lightweight, e.g., stitching and finetuning CLIP ViT-L and SD 3.5 Medium takes only 10 GPU-hours. By lifting powerful pixel-space reward models to latent space, StitchVM opens up a new style of diffusion alignment: instead of rough, yet costly per-sample approximation of the value function, the correct function for the actual, noisy latents is constructed once and then amortized over many samples and iterations. We show that this approach yields improvements across a broad range of downstream steering and post-training methods: DPS becomes 3.2× faster while halving peak GPU memory, and DiffusionNFT becomes 2.3× faster.
Given the wide range of deployment targets, flexible model selection is essential for optimizing performance within a given compute budget. Recent work demonstrates that stitching pretrained models within a model family enables cost-effective interpolation of the accuracy-efficiency tradeoff space. Stitching transforms intermediate activations from one pretrained model into another, producing a new interpolated stitched network. Such networks provide a pool of deployment options along the accuracy-efficiency spectrum. However, existing stitching approaches often yield suboptimal tradeoffs and lack generalizability, as they primarily rely on heuristics to select stitch configurations. We argue that constructing improved accuracy-efficiency tradeoffs requires explicitly capturing and leveraging the similarity between pretrained models being stitched. To this end, we introduce KLAS, a novel stitch selection framework that automates and generalizes stitch selection across model families by leveraging KL divergence between intermediate representations. KLAS identifies the most promising binary stitches from the O(k2n2) possibilities for k pretrained models of depth n. Through comprehensive experiments, we demonstrate that KLAS improves the accuracy-efficiency curve of stitched models at the same finetuning cost as baselines. KLAS achieves up to 1.21% higher ImageNet-1K top-1 accuracy at the same computational cost, or maintains accuracy with a 1.33× reduction in FLOPs.
Debopam Sanyal, Anantharaman Iyer, Alind Khare +5
Georgia Institute of Technology · Microsoft M365 Research · Cisco Research +1
Foundation models have driven rapid progress in computer vision, yet the two dominant paradigms, vision-language foundation models (VLMs) and vision-only foundation models (VFMs), remain only partially compatible. VLMs offer language-grounded semantic alignment but are often visually coarse, while VFMs learn discriminative perceptual geometry but lack semantic grounding. We propose GPUA (Geometry-Preserving Unsupervised Alignment), a framework that integrates the complementary strengths of VFMs and VLMs. Inspired by cross-lingual alignment, GPUA treats VFM features as a visual language and learns an orthogonal mapping that translates the VFM space into the VLM semantic space, preserving geometry and narrowing the modality gap without labels or model parameter updates. GPUA is task-agnostic and requires only feature-level access to pretrained models. Experiments across diverse benchmarks demonstrate improved cross-model compatibility and strong gains in downstream zero-shot recognition and segmentation with negligible overhead. Code is available at https://github.com/Yuteam14/GPUA
Shuwen Yu, Zhanxuan Hu, Yi Zhao +2
Yunnan Normal University, Kunming, China · Kunming University of Science and Technology, Kunming, China