Training-Free Cross-Architecture Merging for Graph Neural Networks

Rishabh Bhattacharya; Vikaskumar Kalsariya; Naresh Manwani

cs.LGFeb 22, 2026

Training-Free Cross-Architecture Merging for Graph Neural Networks

Authors: Rishabh Bhattacharya, Vikaskumar Kalsariya, Naresh Manwani

Organizations: International Institute of Information Technology, Hyderabad, India

Abstract

Model merging has emerged as a powerful paradigm for combining the capabilities of distinct expert models without the high computational cost of retraining, yet current methods are fundamentally constrained to homogeneous architectures. For GNNs, however, message passing is topology-dependent and sensitive to misalignment, making direct parameter-space merging unreliable. To bridge this gap, we introduce H-GRAMA (Heterogeneous Graph Routing and Message Alignment), a training-free framework that lifts merging from parameter space to operator space. We formalize Universal Message Passing Mixture (UMPM), a shared operator family that expresses heterogeneous GNN layers in a common functional language. H-GRAMA enables cross-architecture GNN merging (e.g., GCN to GAT) without retraining, retaining high specialist accuracy in most cases in compatible depth settings and achieving inference speedups of 1.2x to 1.9x over ensembles.

Figures & tables

		Cora			CiteSeer			Actor			Amazon-Ratings			Arxiv
Pair	Depth	64-64	64-128	128-64	64-64	64-128	128-64	64-64	64-128	128-64	64-64	64-128	128-64	64-64	64-128	128-64
GCN–SAGE	2-2	.80 $\pm$ .18	.80 $\pm$ .17	.85 $\pm$ .15	.74 $\pm$ .25	.72 $\pm$ .22	.69 $\pm$ .23	.93 $\pm$ .06	.89 $\pm$ .08	.95 $\pm$ .05	.82 $\pm$ .22	.83 $\pm$ .15	.93 $\pm$ .06	.80 $\pm$ .14	.88 $\pm$ .06	.71 $\pm$ .24
	2-3	.83 $\pm$ .20	.52 $\pm$ .27	.90 $\pm$ .05	.87 $\pm$ .07	.76 $\pm$ .14	.82 $\pm$ .15	.87 $\pm$ .08	.97 $\pm$ .02	.87 $\pm$ .11	.73 $\pm$ .32	.89 $\pm$ .14	.90 $\pm$ .06	.67 $\pm$ .17	.80 $\pm$ .10	.67 $\pm$ .07
	3-2	.78 $\pm$ .23	.85 $\pm$ .13	.65 $\pm$ .20	.63 $\pm$ .26	.75 $\pm$ .20	.75 $\pm$ .18	.76 $\pm$ .12	.71 $\pm$ .18	.73 $\pm$ .12	.97 $\pm$ .06	1.00 $\pm$ .01	.94 $\pm$ .11	.75 $\pm$ .20	.82 $\pm$ .11	.86 $\pm$ .11
GCN–GAT	2-2	.89 $\pm$ .17	.95 $\pm$ .03	.88 $\pm$ .12	.73 $\pm$ .15	.90 $\pm$ .12	.75 $\pm$ .32	.96 $\pm$ .03	.95 $\pm$ .05	.95 $\pm$ .04	.92 $\pm$ .11	.91 $\pm$ .14	.95 $\pm$ .04	.93 $\pm$ .01	.90 $\pm$ .02	.92 $\pm$ .02
	2-3	.94 $\pm$ .06	.77 $\pm$ .11	.94 $\pm$ .03	.85 $\pm$ .16	.88 $\pm$ .04	.86 $\pm$ .11	.96 $\pm$ .03	.95 $\pm$ .06	.80 $\pm$ .10	.90 $\pm$ .11	.92 $\pm$ .06	.96 $\pm$ .05	.81 $\pm$ .09	.84 $\pm$ .06	.81 $\pm$ .09

Table 1: Retention performance across datasets under varying architecture pairs, depths, and widths. Values show minimum retention

\min(\text{Ret}_{A},\text{Ret}_{B})

averaged over 5 seeds. Bold :

\geq 0.80

. Underline :

\geq 0.70

. All merges are training-free and label-free.

Panel of Figure 2 — Figure 2: Retention analysis of the H-GRAMA merging process: Comparison between min-retention strategies for ensemble and parent models versus the full ratio ensemble performance.

	Homogeneous (Same-Arch)		Heterogeneous (Cross-Arch)
Method	Ret A	Ret B	Ret A	Ret B
GRAMA-PMC	0.92 $\pm$ 0.05	0.93 $\pm$ 0.06	—	—
GRAMA-CMC	0.86 $\pm$ 0.08	0.85 $\pm$ 0.12	—	—
GNNMerge++	0.81 $\pm$ 0.29	0.74 $\pm$ 0.13	—	—
GitReBasin	0.83 $\pm$ 0.07	0.81 $\pm$ 0.11	—	—
OTFusion	0.92 $\pm$ 0.07	0.84 $\pm$ 0.06	—	—

Table 2: Comparison of merging methods under homogeneous (same-architecture) and heterogeneous (cross-architecture) settings. Values are reported as mean

\pm

standard deviation. Methods that don’t support heterogenous merging have been denoted with —. Bold indicates best performance per metric group; underline indicates second best.

Pair	Speedup
GAT-GAT	$1.44\pm 0.30$
GAT-GIN	$1.91\pm 1.20$
GAT-SAGE	$1.23\pm 0.31$
GCN-GIN	$1.13\pm 0.11$
GCN-SAGE	$1.32\pm 0.64$
GIN-GIN	$1.31\pm 0.11$

Table 3: Inference speedup of the merged child relative to two-parent ensemble inference on Arxiv.

Figure 3: Phase-wise ablation on Cora (GCN–GIN, 2–3 depth, 64–128 width). Each bar shows the incremental retention gain from adding one pipeline phase.

Panel of Figure 4 — Figure 4: Visualization of absolute basis gate weights $|g_{b}^{\ell}|$ (per layer $\ell$ ) produced by H-GRAMA when merging heterogeneous parents on CiteSeer, using depth $3$ – $3$ in all cases and the following width pairs: (a) GCN–GIN, $128$ – $64$ ; (b) GCN–GraphSAGE, $64$ – $64$ ; (c) GIN–GraphSAGE, $64$ – $64$ .

Figure 5: Loss landscape along the transport-aligned interpolation path on CiteSeer (GCN–GIN, 3–2 depth, 128–64 width). The label-free $\alpha_{\mathrm{auto}}$ (red) closely tracks the true minimum, while $\alpha=0.5$ lies far from the optimal basin.

Figure 6: PCA projection of child pre-activations before (left) and after (right) LFNorm calibration on CiteSeer (GCN–SAGE, 2–2 depth, 64–64 width). Centroid distance contracts by a substantial amount, aligning the child with interpolated parent targets.

Explore similar work

GFFMERGE: Efficient Merging of Graph Neural Force Fields and Beyond
Jun 2, 2026Parth Verma, Parv P. Singh, Vipul Garg +3Graph Neural Networks Machine Learning Interatomic Potentials
Department of Computer Science, Indian Institute of Technology Delhi, Hauz Khas, New Delhi, India · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi, Hauz Khas, New Delhi, India · Department of Civil Engineering, Indian Institute of Technology Delhi, Hauz Khas, New Delhi, India
CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformations
Oct 1, 2026Adir Dayan, Yam Eitan, Haggai MaronModel Compression Hypernetworks
Technion – Israel Institute of Technology · NVIDIA Research
Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion
Sep 19, 2026Aditya Pola, Vineeth N. BalasubramanianModel Merging
IIT Hyderabad · Microsoft Research, India

Jun 2, 2026J/K move · Enter open · S save

GFFMERGE: Efficient Merging of Graph Neural Force Fields and Beyond

Parth Verma, Parv P. Singh, Vipul Garg, Ishita Thakre+2

Department of Computer Science, Indian Institute of Technology Delhi, Hauz Khas, New Delhi, India · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi, Hauz Khas, New Delhi, India · Department of Civil Engineering, Indian Institute of Technology Delhi, Hauz Khas, New Delhi, India

Graph Neural Networks Machine Learning Interatomic Potentials Message Passing Neural Networks Model Merging

Open paper AskPDF arXiv

Graph Neural Networks (GNNs) have revolutionized Neural Force Fields for atomistic simulations, achieving near-quantum accuracy at reduced cost, yet adapting these models to new chemical systems requires expensive retraining of foundation models. Inspired by model merging in vision and language processing, we introduce GFFMERGE, the first principled framework for closed-form model merging in GNNs. We exploit the linear structure of message-passing layers and formulate merging as a convex embedding-alignment problem with an analytical solution. Through the first systematic benchmarking of model merging for GNNs, we show that existing methods designed for vision and language catastrophically fail on force field regression, while GFFMERGE recovers performance approaching gold standard joint training. Across molecular (MD17, MD22), solid-state (LiPS20), and large-scale graph benchmarks, GFFMERGE and GNNMERGE (its generic GNN counterpart) achieve 5-27

\times

speedups while enabling modular composition of specialized models. Remarkably, our closed-form solution alone outperforms all baseline methods before fine-tuning and provides superior initialization for faster, data-efficient convergence.

Abstract

Explore similar work

GFFMERGE: Efficient Merging of Graph Neural Force Fields and Beyond

CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformations

Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion