SuperAI: Merging Capabilities Across Heterogeneous Foundation Models

Jingtong Dou; Chuancheng Shi; Wenhua Wu; Wei Bao; Fei Shen; Zhiyong Wang; Tat-Seng Chua

cs.AIOct 11, 2026

SuperAI: Merging Capabilities Across Heterogeneous Foundation Models

Authors: Jingtong Dou, Chuancheng Shi, Wenhua Wu, Wei Bao, Fei Shen, Zhiyong Wang, Tat-Seng Chua

Organizations: University of Sydney Sydney, Australia · National University of Singapore Singapore

Abstract

Enabling the reuse and composition of knowledge and capabilities distributed across foundation models is an important step toward building scalable artificial intelligence (AI) systems. However, architectural differences and incompatible representations make it difficult to transfer and integrate these capabilities through a unified cross-model interface. To address this challenge, we propose SuperAI, a cross-architecture multi-model fusion framework that integrates complementary capabilities from multiple source models into a host model through orthogonal alignment of low-dimensional task-relevant representations. Specifically, we first extract task-relevant representations from the source models and the host model using shared anchor samples, and apply principal component analysis (PCA) to construct low-dimensional functional subspaces with the same dimensionality. Next, we use orthogonal Procrustes mappings to align the source and host coordinate systems, establishing cross-model correspondences while preserving the internal geometry of each source subspace. Finally, we train a lightweight residual multilayer perceptron (MLP) with joint supervision on task outputs and source-model increments, learning input-dependent residual increments that are injected into the host model for multi-model capability fusion. Experiments on three host models across 15 capability domains show that SuperAI achieves higher average domain scores than the evaluated cross-architecture fusion methods and better preserves general capabilities as the number of source models increases. Ablation studies further support the effectiveness of orthogonal alignment, nonlinear residual mapping, and joint supervision.

Figures & tables

Figure 1: Conventional Model Merging Versus SuperAI. Conventional parameter merging requires compatible parameter spaces, whereas SuperAI integrates heterogeneous source capabilities into a merged model through orthogonal subspace alignment and lightweight residual fusion.

Figure 2: Overview of Proposed SuperAI Framework. Shared prompts are used to align source and host representations through PCA and orthogonal rotation. All layer pairs are evaluated to select one shared host layer and the corresponding source layers. The aligned source features define capability increments

\Delta_{e}

, which supervise a shared residual MLP together with task and domain objectives. At inference, only the frozen host model and the trained MLP are required.

Figure 3: Cross-Model Representations Before and After Orthogonal Alignment. Source and host representations differ substantially before alignment and become closely aligned afterward.

Figure 4: Cross-model Alignment Evaluation. Match, CKA, and Retrieval@1 scores across 15 source models with Llama and Qwen as host models.

Host model: Llama-3.2-3B
Method	Math	Safety	Law	Phys.	Sci.	C.Sense	Social	Code	Multi.	Fin./Bus.	Astro.	Med.	Climate	Cyber	Chem.	Avg.
Domain	67.7	72.0	46.5	80.9	76.5	81.6	75.2	81.5	69.4	15.0	74.3	58.4	71.4	36.2	49.0	63.7
Original	16.7	56.8	38.0	68.3	67.9	66.2	58.7	39.7	56.1	3.7	60.5	50.9	42.3	31.4	44.0	46.7
THESEUS	24.1	57.3	40.3	62.9	63.5	60.0	53.8	42.1	44.0	12.7	54.2	40.6	38.4	34.7	32.1	44.0 ( $\downarrow 2.7$ )
T&M	17.6	59.5	41.2	61.4	64.8	61.7	54.2	41.7	44.3	10.9	49.7	40.3	35.8	31.2	43.4	43.8 ( $\downarrow 2.9$ )
SuperAI	40.7	70.9	40.8	89.1	69.8	68.2	63.3	28.3	55.0	14.7	57.9	48.2	72.5	48.9	44.0	54.2 ( $\uparrow 7.5$ )

Table 1: Quantitative Comparison of Multi-source Capability Fusion Across Different Host Models. We report the original host performance and multi-source fusion results across 15 capability domains. The best and second-best results among the fusion methods are shown in bold and underlined , respectively. Green and red arrows in the Avg. column denote gains and decreases relative to the corresponding original host.

Figure 5: General capability retention. MMLU retention of THESEUS, T&M, and SuperAI with 1–15 source models.

15-domain average score
Objective	Llama-3.2-3B	Qwen2.5-3B	Pythia-2.8B
Original	46.7	56.2	22.9
$\lambda_{\mathrm{inc}}=0$	50.6 ( $\uparrow 3.9$ )	57.5 ( $\uparrow 1.3$ )	27.1 ( $\uparrow 4.2$ )
$\lambda_{\mathrm{task}}=0$	44.8 ( $\downarrow 1.9$ )	56.3 ( $\uparrow 0.1$ )	24.0 ( $\uparrow 1.1$ )
Joint	54.2 ( $\uparrow 7.5$ )	61.3 ( $\uparrow 5.1$ )	32.8 ( $\uparrow 9.9$ )
Cosine-similarity gain

Table 2: Effect of Training Objectives. We report average scores across 15 domains and cosine-similarity gains for three host models. For each host and metric, the best and second-best results are bolded and underlined, respectively.

Mapping Method	Llama-3.2-3B	Qwen2.5-3B	Pythia-2.8B
Original	46.7	56.2	22.9
No Alignment	47.6 ( $\uparrow 0.9$ )	56.1 ( $\downarrow 0.1$ )	21.0 ( $\downarrow 1.9$ )
PCA Only	47.8 ( $\uparrow 1.1$ )	56.9 ( $\uparrow 0.7$ )	28.6 ( $\uparrow 5.7$ )
PCA + Linear	53.2 ( $\uparrow 6.5$ )	57.5 ( $\uparrow 1.3$ )	27.2 ( $\uparrow 4.3$ )
PCA + Orthogonal	54.2 ( $\uparrow 7.5$ )	61.3 ( $\uparrow 5.1$ )	32.8 ( $\uparrow 9.9$ )

Table 3: Effect of Orthogonal Subspace Alignment. We report 15-domain average scores for three host models, with changes relative to each original host in parentheses.

Mapper	Llama-3.2-3B	Qwen2.5-3B	Pythia-2.8B
Original	46.7	56.2	22.9
Linear	50.6 ( $\uparrow 3.9$ )	56.0 ( $\downarrow 0.2$ )	30.1 ( $\uparrow 7.2$ )
MLP	54.2 ( $\uparrow 7.5$ )	61.3 ( $\uparrow 5.1$ )	32.8 ( $\uparrow 9.9$ )

Table 4: Effect of Residual Mapper Architecture. We report average scores across 15 domains for three host models. Parentheses indicate score changes relative to the original host.

Figure 6: Sensitivity to Residual Scaling. Average scores across 15 domains for three host models as the residual scaling factor

\alpha

varies from 0 to 2.0.

15-domain average score
Layer Pair	Llama-3.2-3B	Qwen2.5-3B	Pythia-2.8B
Original	46.7	56.2	22.9
Weakly Matched	51.1 ( $\uparrow 4.4$ )	59.0 ( $\uparrow 2.8$ )	28.4 ( $\uparrow 5.5$ )
Medium Matched	53.5 ( $\uparrow 6.8$ )	60.4 ( $\uparrow 4.2$ )	31.6 ( $\uparrow 8.7$ )
Best Matched	54.2 ( $\uparrow 7.5$ )	61.3 ( $\uparrow 5.1$ )	32.8 ( $\uparrow 9.9$ )
Post-alignment Match score

Table 5: Effect of Layer-pair Alignment Quality. We report 15-domain averages and post-alignment Match scores; parentheses show changes from the baseline. Best and second-best results are bolded and underlined.

Figure 7: Effect of Host Model Scale on Multi-source Fusion Capacity. Bars show mean 15-domain gains at the maximum stable fusion capacity

k^{\ast}

, with error bars indicating standard deviations across three seeds. Lines show

k^{\ast}

; 3/3 denotes positive gains in all three seeds. Missing entries indicate unavailable model scales.

Host Model	Math	Safety	Law	Physics	Avg.
Qwen2.5-1.5B	14.3 ( $\downarrow 12.3$ )	72.5 ( $\uparrow 8.0$ )	49.3 ( $\uparrow 8.5$ )	94.0 ( $\uparrow 14.1$ )	57.5 ( $\uparrow 4.6$ )
Qwen2.5-3B	45.7 ( $\uparrow 19.4$ )	76.6 ( $\uparrow 15.3$ )	53.5 ( $\uparrow 14.1$ )	94.9 ( $\uparrow 9.8$ )	67.7 ( $\uparrow 14.6$ )
Qwen2.5-7B	57.3 ( $\uparrow 40.7$ )	79.3 ( $\uparrow 3.3$ )	56.3 ( $\uparrow 8.5$ )	95.7 ( $\uparrow 4.3$ )	72.2 ( $\uparrow 14.2$ )
Qwen2.5-14B	57.7 ( $\uparrow 44.0$ )	80.2 ( $\uparrow 8.9$ )	59.2 ( $\downarrow 5.6$ )	97.9 ( $\uparrow 1.1$ )	73.8 ( $\uparrow 12.1$ )

Table 6: Domain-wise Capability Absorption Across Qwen2.5 Scales. Each cell reports the post-fusion score and its change from the corresponding base model. Avg. denotes the four-domain average; best and second-best results are bolded and underlined.

Figure 8: Generalization Across Host Architectures and Scales. Evaluation of SuperAI in the Climate, Science, Common Sense, and Social domains: (a) generalization to unseen datasets across host architectures; (b) generalization performance across host model scales.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Source Model	Training Dataset	Task Metric $\uparrow$
Science	ScienceQA-OFT-8B	ARC-Challenge	Accuracy
Common Sense	CommonsenseQA-OFT-8B	CommonsenseQA	Accuracy
Social	Social-IQA-OFT-8B	BIG-bench Social IQA	Accuracy
Math	NuminaMath-OFT-8B	SVAMP	Numeric Answer Accuracy
Code	Qwen2.5-Coder-7B-Instruct	MBPP	Pass@1
Multilingual	Llama-3.2-3B Multilingual	M-ARC	Averaged Accuracy

Appendix

Table 7: Training datasets and evaluation metrics for the 15 capability domains. We list the source models, training datasets, and corresponding evaluation metrics. All metrics are higher-is-better.

Symbol	Definition
Models and representations
$f_{h},\{f_{s}\}_{s=1}^{K}$	Host model and $K$ source models.
$d_{h},d_{s}$	Hidden dimensions of the host and source model $s$ .
$\mathcal{A}_{s}=\{a_{i}\}_{i=1}^{n_{s}}$	Shared anchor prompts for source model $s$ and the host.
$\ell_{m},h_{m}^{(\ell_{m})}(a_{i})$	Layer index and last non-padding-token hidden state of model $m\in\{s,h\}$ .
$H_{m}^{(\ell_{m})}$	Matrix of hidden states for prompts in $\mathcal{A}_{s}$ .

Appendix

Table 8: Summary of notation used in SuperAI.

PCA Rank	Llama-3.2-3B	Qwen2.5-3B	Pythia-2.8B
Base	46.7	56.2	22.9
16	56.3 ( $\uparrow 9.6$ )	60.2 ( $\uparrow 4.0$ )	28.7 ( $\uparrow 5.8$ )
32	55.3 ( $\uparrow 8.6$ )	61.3 ( $\uparrow 5.1$ )	28.3 ( $\uparrow 5.4$ )
64	54.5 ( $\uparrow 7.8$ )	61.2 ( $\uparrow 5.0$ )	29.0 ( $\uparrow 6.1$ )
128	54.2 ( $\uparrow 7.5$ )	61.3 ( $\uparrow 5.1$ )	32.8 ( $\uparrow 9.9$ )
256	55.2 ( $\uparrow 8.5$ )	60.8 ( $\uparrow 4.6$ )	28.9 ( $\uparrow 6.0$ )

Appendix

Table 9: Effect of PCA subspace dimensionality. Each cell reports the 15-domain average score, with the absolute change from the corresponding base model shown in parentheses. PCA rank denotes the requested upper bound.

Examples/Domain	Llama-3.2-3B	Qwen2.5-3B	Pythia-2.8B
Base	46.7	56.2	22.9
50	54.1 ( $\uparrow 7.4$ )	57.9 ( $\uparrow 1.7$ )	27.8 ( $\uparrow 4.9$ )
100	56.3 ( $\uparrow 9.6$ )	59.6 ( $\uparrow 3.4$ )	27.1 ( $\uparrow 4.2$ )
200	57.6 ( $\uparrow 10.9$ )	58.9 ( $\uparrow 2.7$ )	26.9 ( $\uparrow 4.0$ )
500	55.6 ( $\uparrow 8.9$ )	60.0 ( $\uparrow 3.8$ )	27.4 ( $\uparrow 4.5$ )
1000	54.2 ( $\uparrow 7.5$ )	61.3 ( $\uparrow 5.1$ )	32.8 ( $\uparrow 9.9$ )

Appendix

Table 10: Effect of training data volume. Each cell reports the 15-domain average score, with the absolute change from the corresponding base model shown in parentheses. The training cap is applied per domain; domains with fewer examples use all available training samples.

Figure 9: Multi-source scalability across capability domains. Capability scores of SuperAI, THESEUS, and T&M as the number of merged sources increases across Math, Safety, Law, and Physics.

Orthogonal Procrustes Alignment

See all 11 papers

Jun 3, 2026

Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models

Foundation models have driven rapid progress in computer vision, yet the two dominant paradigms, vision-language foundation models (VLMs) and vision-only foundation models (VFMs), remain only partially compatible. VLMs offer language-grounded semantic alignment but are often visually coarse, while VFMs learn discriminative perceptual geometry but lack semantic grounding. We propose GPUA (Geometry-Preserving Unsupervised Alignment), a framework that integrates the complementary strengths of VFMs and VLMs. Inspired by cross-lingual alignment, GPUA treats VFM features as a visual language and learns an orthogonal mapping that translates the VFM space into the VLM semantic space, preserving geometry and narrowing the modality gap without labels or model parameter updates. GPUA is task-agnostic and requires only feature-level access to pretrained models. Experiments across diverse benchmarks demonstrate improved cross-model compatibility and strong gains in downstream zero-shot recognition and segmentation with negligible overhead. Code is available at https://github.com/Yuteam14/GPUA

Shuwen Yu, Zhanxuan Hu, Yi Zhao +2

Yunnan Normal University, Kunming, China · Kunming University of Science and Technology, Kunming, China

Cross-Modal Representation Learning Vision Foundation Model Adaptation Cross-Modal Feature Matching Multimodal Foundation Models Vision-Language Alignment

Ask PDF

Feb 14, 2026

StackingNet: Collective Inference Across Independent AI Foundation Models

Artificial intelligence built on large foundation models has transformed language understanding, computer vision, and reasoning, yet these systems remain isolated and cannot readily share their capabilities. Coordinating the complementary strengths of independently developed, black-box foundation models is essential for trustworthy intelligent systems, yet no established method exists. Here we show that such coordination can be achieved through a meta-ensemble framework termed StackingNet, which aggregates the output predictions of independent models at inference. StackingNet improves accuracy, reduces individual-model error and group-wise disparities, ranks model reliability, and identifies or prunes models that degrade performance, all without access to internal parameters or training data. Across language comprehension, visual attribute estimation, and academic paper rating, it consistently outperforms individual models and classic ensembles, with gains that persist when the base models are uniformly strong. These gains stem from variance reduction and consensus alignment among independent models rather than from any emergent group cognition, and they widen as the model pool grows more diverse. By turning model diversity from a source of inconsistency into a resource for cooperation, StackingNet offers a practical path toward coordinated artificial intelligence, where progress emerges not only from larger single models but from principled cooperation among many specialized ones.

Siyang Li, Chenhao Liu, Dongrui Wu +2

School of Artiﬁcial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China. · School of Civil and Hydraulic Engineering, Huazhong University of Science and Technology, Wuhan 430074, China.

Ensemble Learning Language Model Ensembles Stacked Ensemble Learning

Ask PDF

Oct 13, 2026

To Align or Not To Align: Check Your COMPASS Before You Train

Combining pretrained models is a common strategy for building AI systems, whether by aligning a vision encoder with a language model or merging fine-tuned checkpoints. However, finding compatible pairs typically requires costly empirical search, as compatibility is known only after alignment or merging. We propose COMPASS (COMPatibility ASsessment Score), a theoretically motivated diagnostic that estimates whether a model pair will be compatible prior to alignment or merging. Building on the observation that representational convergence is most robust at the level of local neighborhoods, COMPASS decomposes model compatibility into two cheap-to-compute axes: a pair-specific ease, defined as the

k

-NN overlap between the two encoders, and a paradigm-level ceiling, capturing the maximum overlap their training paradigms allow. Together, these two axes provide a ranking of candidate model pairs and prescribe how to act on each pair: (i) prioritize pairs that already agree and whose paradigms are compatible, (ii) invest additional compute on pairs with compatible paradigms but low initial agreement, (iii) switch the pretraining paradigm when it cannot support sufficient agreement, or (iv) deprioritize pairs that score poorly on both axes. We apply COMPASS to

72

cross-modal alignment experiments and

392

task-arithmetic model merges and show that compatibility estimates by COMPASS strongly correlate with downstream success, achieving average Pearson correlation of

0.91

on cross-modal alignment tasks and

0.70

on model merging.

Fabian Gröger, Petar Damjanović, Maria Brbić

EPFL · University of Basel · HSLU

Language Model Merging Cross-Modal Alignment

Ask PDF

Abstract

Orthogonal Procrustes Alignment

Explore similar work

Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models

StackingNet: Collective Inference Across Independent AI Foundation Models

To Align or Not To Align: Check Your COMPASS Before You Train