SuperAI: Merging Capabilities Across Heterogeneous Foundation Models

Jingtong Dou; Chuancheng Shi; Wenhua Wu; Wei Bao; Fei Shen; Zhiyong Wang; Tat-Seng Chua

cs.AIOct 11, 2026

SuperAI: Merging Capabilities Across Heterogeneous Foundation Models

Authors: Jingtong Dou, Chuancheng Shi, Wenhua Wu, Wei Bao, Fei Shen, Zhiyong Wang, Tat-Seng Chua

Organizations: University of Sydney Sydney, Australia · National University of Singapore Singapore

Abstract

Enabling the reuse and composition of knowledge and capabilities distributed across foundation models is an important step toward building scalable artificial intelligence (AI) systems. However, architectural differences and incompatible representations make it difficult to transfer and integrate these capabilities through a unified cross-model interface. To address this challenge, we propose SuperAI, a cross-architecture multi-model fusion framework that integrates complementary capabilities from multiple source models into a host model through orthogonal alignment of low-dimensional task-relevant representations. Specifically, we first extract task-relevant representations from the source models and the host model using shared anchor samples, and apply principal component analysis (PCA) to construct low-dimensional functional subspaces with the same dimensionality. Next, we use orthogonal Procrustes mappings to align the source and host coordinate systems, establishing cross-model correspondences while preserving the internal geometry of each source subspace. Finally, we train a lightweight residual multilayer perceptron (MLP) with joint supervision on task outputs and source-model increments, learning input-dependent residual increments that are injected into the host model for multi-model capability fusion. Experiments on three host models across 15 capability domains show that SuperAI achieves higher average domain scores than the evaluated cross-architecture fusion methods and better preserves general capabilities as the number of source models increases. Ablation studies further support the effectiveness of orthogonal alignment, nonlinear residual mapping, and joint supervision.

Figures & tables

Figure 1: Conventional Model Merging Versus SuperAI. Conventional parameter merging requires compatible parameter spaces, whereas SuperAI integrates heterogeneous source capabilities into a merged model through orthogonal subspace alignment and lightweight residual fusion.

Figure 2: Overview of Proposed SuperAI Framework. Shared prompts are used to align source and host representations through PCA and orthogonal rotation. All layer pairs are evaluated to select one shared host layer and the corresponding source layers. The aligned source features define capability increments

\Delta_{e}

, which supervise a shared residual MLP together with task and domain objectives. At inference, only the frozen host model and the trained MLP are required.

Figure 3: Cross-Model Representations Before and After Orthogonal Alignment. Source and host representations differ substantially before alignment and become closely aligned afterward.

Figure 4: Cross-model Alignment Evaluation. Match, CKA, and Retrieval@1 scores across 15 source models with Llama and Qwen as host models.

Host model: Llama-3.2-3B
Method	Math	Safety	Law	Phys.	Sci.	C.Sense	Social	Code	Multi.	Fin./Bus.	Astro.	Med.	Climate	Cyber	Chem.	Avg.
Domain	67.7	72.0	46.5	80.9	76.5	81.6	75.2	81.5	69.4	15.0	74.3	58.4	71.4	36.2	49.0	63.7
Original	16.7	56.8	38.0	68.3	67.9	66.2	58.7	39.7	56.1	3.7	60.5	50.9	42.3	31.4	44.0	46.7
THESEUS	24.1	57.3	40.3	62.9	63.5	60.0	53.8	42.1	44.0	12.7	54.2	40.6	38.4	34.7	32.1	44.0 ( $\downarrow 2.7$ )
T&M	17.6	59.5	41.2	61.4	64.8	61.7	54.2	41.7	44.3	10.9	49.7	40.3	35.8	31.2	43.4	43.8 ( $\downarrow 2.9$ )
SuperAI	40.7	70.9	40.8	89.1	69.8	68.2	63.3	28.3	55.0	14.7	57.9	48.2	72.5	48.9	44.0	54.2 ( $\uparrow 7.5$ )

Table 1: Quantitative Comparison of Multi-source Capability Fusion Across Different Host Models. We report the original host performance and multi-source fusion results across 15 capability domains. The best and second-best results among the fusion methods are shown in bold and underlined , respectively. Green and red arrows in the Avg. column denote gains and decreases relative to the corresponding original host.

Figure 5: General capability retention. MMLU retention of THESEUS, T&M, and SuperAI with 1–15 source models.

15-domain average score
Objective	Llama-3.2-3B	Qwen2.5-3B	Pythia-2.8B
Original	46.7	56.2	22.9
$\lambda_{\mathrm{inc}}=0$	50.6 ( $\uparrow 3.9$ )	57.5 ( $\uparrow 1.3$ )	27.1 ( $\uparrow 4.2$ )
$\lambda_{\mathrm{task}}=0$	44.8 ( $\downarrow 1.9$ )	56.3 ( $\uparrow 0.1$ )	24.0 ( $\uparrow 1.1$ )
Joint	54.2 ( $\uparrow 7.5$ )	61.3 ( $\uparrow 5.1$ )	32.8 ( $\uparrow 9.9$ )
Cosine-similarity gain

Table 2: Effect of Training Objectives. We report average scores across 15 domains and cosine-similarity gains for three host models. For each host and metric, the best and second-best results are bolded and underlined, respectively.

Mapping Method	Llama-3.2-3B	Qwen2.5-3B	Pythia-2.8B
Original	46.7	56.2	22.9
No Alignment	47.6 ( $\uparrow 0.9$ )	56.1 ( $\downarrow 0.1$ )	21.0 ( $\downarrow 1.9$ )
PCA Only	47.8 ( $\uparrow 1.1$ )	56.9 ( $\uparrow 0.7$ )	28.6 ( $\uparrow 5.7$ )
PCA + Linear	53.2 ( $\uparrow 6.5$ )	57.5 ( $\uparrow 1.3$ )	27.2 ( $\uparrow 4.3$ )
PCA + Orthogonal	54.2 ( $\uparrow 7.5$ )	61.3 ( $\uparrow 5.1$ )	32.8 ( $\uparrow 9.9$ )

Table 3: Effect of Orthogonal Subspace Alignment. We report 15-domain average scores for three host models, with changes relative to each original host in parentheses.

Mapper	Llama-3.2-3B	Qwen2.5-3B	Pythia-2.8B
Original	46.7	56.2	22.9
Linear	50.6 ( $\uparrow 3.9$ )	56.0 ( $\downarrow 0.2$ )	30.1 ( $\uparrow 7.2$ )
MLP	54.2 ( $\uparrow 7.5$ )	61.3 ( $\uparrow 5.1$ )	32.8 ( $\uparrow 9.9$ )

Table 4: Effect of Residual Mapper Architecture. We report average scores across 15 domains for three host models. Parentheses indicate score changes relative to the original host.

Figure 6: Sensitivity to Residual Scaling. Average scores across 15 domains for three host models as the residual scaling factor

\alpha

varies from 0 to 2.0.

15-domain average score
Layer Pair	Llama-3.2-3B	Qwen2.5-3B	Pythia-2.8B
Original	46.7	56.2	22.9
Weakly Matched	51.1 ( $\uparrow 4.4$ )	59.0 ( $\uparrow 2.8$ )	28.4 ( $\uparrow 5.5$ )
Medium Matched	53.5 ( $\uparrow 6.8$ )	60.4 ( $\uparrow 4.2$ )	31.6 ( $\uparrow 8.7$ )
Best Matched	54.2 ( $\uparrow 7.5$ )	61.3 ( $\uparrow 5.1$ )	32.8 ( $\uparrow 9.9$ )
Post-alignment Match score

Table 5: Effect of Layer-pair Alignment Quality. We report 15-domain averages and post-alignment Match scores; parentheses show changes from the baseline. Best and second-best results are bolded and underlined.

Figure 7: Effect of Host Model Scale on Multi-source Fusion Capacity. Bars show mean 15-domain gains at the maximum stable fusion capacity

k^{\ast}

, with error bars indicating standard deviations across three seeds. Lines show

k^{\ast}

; 3/3 denotes positive gains in all three seeds. Missing entries indicate unavailable model scales.

Host Model	Math	Safety	Law	Physics	Avg.
Qwen2.5-1.5B	14.3 ( $\downarrow 12.3$ )	72.5 ( $\uparrow 8.0$ )	49.3 ( $\uparrow 8.5$ )	94.0 ( $\uparrow 14.1$ )	57.5 ( $\uparrow 4.6$ )
Qwen2.5-3B	45.7 ( $\uparrow 19.4$ )	76.6 ( $\uparrow 15.3$ )	53.5 ( $\uparrow 14.1$ )	94.9 ( $\uparrow 9.8$ )	67.7 ( $\uparrow 14.6$ )
Qwen2.5-7B	57.3 ( $\uparrow 40.7$ )	79.3 ( $\uparrow 3.3$ )	56.3 ( $\uparrow 8.5$ )	95.7 ( $\uparrow 4.3$ )	72.2 ( $\uparrow 14.2$ )
Qwen2.5-14B	57.7 ( $\uparrow 44.0$ )	80.2 ( $\uparrow 8.9$ )	59.2 ( $\downarrow 5.6$ )	97.9 ( $\uparrow 1.1$ )	73.8 ( $\uparrow 12.1$ )

Table 6: Domain-wise Capability Absorption Across Qwen2.5 Scales. Each cell reports the post-fusion score and its change from the corresponding base model. Avg. denotes the four-domain average; best and second-best results are bolded and underlined.

Figure 8: Generalization Across Host Architectures and Scales. Evaluation of SuperAI in the Climate, Science, Common Sense, and Social domains: (a) generalization to unseen datasets across host architectures; (b) generalization performance across host model scales.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Source Model	Training Dataset	Task Metric $\uparrow$
Science	ScienceQA-OFT-8B	ARC-Challenge	Accuracy
Common Sense	CommonsenseQA-OFT-8B	CommonsenseQA	Accuracy
Social	Social-IQA-OFT-8B	BIG-bench Social IQA	Accuracy
Math	NuminaMath-OFT-8B	SVAMP	Numeric Answer Accuracy
Code	Qwen2.5-Coder-7B-Instruct	MBPP	Pass@1
Multilingual	Llama-3.2-3B Multilingual	M-ARC	Averaged Accuracy

Appendix

Table 7: Training datasets and evaluation metrics for the 15 capability domains. We list the source models, training datasets, and corresponding evaluation metrics. All metrics are higher-is-better.

Symbol	Definition
Models and representations
$f_{h},\{f_{s}\}_{s=1}^{K}$	Host model and $K$ source models.
$d_{h},d_{s}$	Hidden dimensions of the host and source model $s$ .
$\mathcal{A}_{s}=\{a_{i}\}_{i=1}^{n_{s}}$	Shared anchor prompts for source model $s$ and the host.
$\ell_{m},h_{m}^{(\ell_{m})}(a_{i})$	Layer index and last non-padding-token hidden state of model $m\in\{s,h\}$ .
$H_{m}^{(\ell_{m})}$	Matrix of hidden states for prompts in $\mathcal{A}_{s}$ .

Appendix

Table 8: Summary of notation used in SuperAI.

PCA Rank	Llama-3.2-3B	Qwen2.5-3B	Pythia-2.8B
Base	46.7	56.2	22.9
16	56.3 ( $\uparrow 9.6$ )	60.2 ( $\uparrow 4.0$ )	28.7 ( $\uparrow 5.8$ )
32	55.3 ( $\uparrow 8.6$ )	61.3 ( $\uparrow 5.1$ )	28.3 ( $\uparrow 5.4$ )
64	54.5 ( $\uparrow 7.8$ )	61.2 ( $\uparrow 5.0$ )	29.0 ( $\uparrow 6.1$ )
128	54.2 ( $\uparrow 7.5$ )	61.3 ( $\uparrow 5.1$ )	32.8 ( $\uparrow 9.9$ )
256	55.2 ( $\uparrow 8.5$ )	60.8 ( $\uparrow 4.6$ )	28.9 ( $\uparrow 6.0$ )

Appendix

Table 9: Effect of PCA subspace dimensionality. Each cell reports the 15-domain average score, with the absolute change from the corresponding base model shown in parentheses. PCA rank denotes the requested upper bound.

Examples/Domain	Llama-3.2-3B	Qwen2.5-3B	Pythia-2.8B
Base	46.7	56.2	22.9
50	54.1 ( $\uparrow 7.4$ )	57.9 ( $\uparrow 1.7$ )	27.8 ( $\uparrow 4.9$ )
100	56.3 ( $\uparrow 9.6$ )	59.6 ( $\uparrow 3.4$ )	27.1 ( $\uparrow 4.2$ )
200	57.6 ( $\uparrow 10.9$ )	58.9 ( $\uparrow 2.7$ )	26.9 ( $\uparrow 4.0$ )
500	55.6 ( $\uparrow 8.9$ )	60.0 ( $\uparrow 3.8$ )	27.4 ( $\uparrow 4.5$ )
1000	54.2 ( $\uparrow 7.5$ )	61.3 ( $\uparrow 5.1$ )	32.8 ( $\uparrow 9.9$ )

Appendix

Table 10: Effect of training data volume. Each cell reports the 15-domain average score, with the absolute change from the corresponding base model shown in parentheses. The training cap is applied per domain; domains with fewer examples use all available training samples.

Figure 9: Multi-source scalability across capability domains. Capability scores of SuperAI, THESEUS, and T&M as the number of merged sources increases across Math, Safety, Law, and Physics.

Orthogonal Procrustes Alignment

See all 11 papers

Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models
Jun 3, 2026Shuwen Yu, Zhanxuan Hu, Yi Zhao +2Cross-Modal Representation Learning Vision Foundation Model Adaptation
Yunnan Normal University, Kunming, China · Kunming University of Science and Technology, Kunming, China
StackingNet: Collective Inference Across Independent AI Foundation Models
Feb 14, 2026Siyang Li, Chenhao Liu, Dongrui Wu +2Ensemble Learning Language Model Ensembles
School of Artiﬁcial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China. · School of Civil and Hydraulic Engineering, Huazhong University of Science and Technology, Wuhan 430074, China.
To Align or Not To Align: Check Your COMPASS Before You Train
Oct 13, 2026Fabian Gröger, Petar Damjanović, Maria BrbićLanguage Model Merging Cross-Modal Alignment
EPFL · University of Basel · HSLU

Jun 3, 2026J/K move · Enter open · S save

Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models

Shuwen Yu, Zhanxuan Hu, Yi Zhao, Yonghang Tai+1

Yunnan Normal University, Kunming, China · Kunming University of Science and Technology, Kunming, China

Cross-Modal Representation Learning Vision Foundation Model Adaptation Cross-Modal Feature Matching Multimodal Foundation Models Vision-Language Alignment

Open paper AskPDF arXiv

Foundation models have driven rapid progress in computer vision, yet the two dominant paradigms, vision-language foundation models (VLMs) and vision-only foundation models (VFMs), remain only partially compatible. VLMs offer language-grounded semantic alignment but are often visually coarse, while VFMs learn discriminative perceptual geometry but lack semantic grounding. We propose GPUA (Geometry-Preserving Unsupervised Alignment), a framework that integrates the complementary strengths of VFMs and VLMs. Inspired by cross-lingual alignment, GPUA treats VFM features as a visual language and learns an orthogonal mapping that translates the VFM space into the VLM semantic space, preserving geometry and narrowing the modality gap without labels or model parameter updates. GPUA is task-agnostic and requires only feature-level access to pretrained models. Experiments across diverse benchmarks demonstrate improved cross-model compatibility and strong gains in downstream zero-shot recognition and segmentation with negligible overhead. Code is available at https://github.com/Yuteam14/GPUA

Abstract

Orthogonal Procrustes Alignment

Explore similar work

Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models

StackingNet: Collective Inference Across Independent AI Foundation Models

To Align or Not To Align: Check Your COMPASS Before You Train