Transferable Graph Metanetworks

Yuxin Ma; Adir Dayan; Yam Eitan; Haggai Maron; Soledad Villar

stat.MLSep 30, 2026

Transferable Graph Metanetworks

Authors: Yuxin Ma, Adir Dayan, Yam Eitan, Haggai Maron, Soledad Villar

Organizations: Department of Applied Mathematics and Statistics, Johns Hopkins University · Technion · NVIDIA

Abstract

A weight space network (or metanetwork) takes the weights of another neural network as input and predicts properties of it. Most prior work trains such models on input networks of one or a few fixed sizes and evaluates them in-distribution. The few attempts at out-of-distribution size generalization remain limited in scope and have achieved only modest success. Consequently, the potential efficiency gains of training on small networks and evaluating on much larger ones remain largely unrealized. We propose Transferable Graph Metanetworks, which extend the graph metanetwork paradigm with a set of modifications that make performance transferable across input networks of different widths. The modifications follow two principles: invariance to the ways in which networks of different widths represent the same function, and continuity, such that weights representing similar functions receive similar predictions. We further study whether size generalization is possible for input networks trained independently from random initialization. Empirically, our modifications significantly improve size generalization on every task we consider. Performance is strongest on input networks trained under the maximal-update parameterization ( $μ$ P), where it remains robust up to $42\times$ the training width. Theoretically, we explain these observations with infinite-width limit theory: we prove size-generalization guarantees for our model on $μ$ P-trained inputs, and explain why it can fail under other parameterizations.

Figures & tables

Panel of Figure 1 — Figure 1 : Function-preserving widenings of an MLP. (b) Copying and rescaling weights duplicates hidden neurons, as in (a), so the function is unchanged; duplication-compatible GMN is invariant to this transformation. (c) More generally, the copied weights can be redistributed across rows of the block, which still amounts to the duplication of neurons in (a) and preserves the function; matrix-product GMN is invariant to this transformation.

Panel of Figure 2 — Figure 2 : Size generalization with independently trained weights, comparing the baseline GMN, the duplication-compatible GMN, and the matrix-product GMN on SP- and $\mu$ P-trained input networks, for two baseline architectures. Each panel trains at the smallest width and evaluates at the sequence of larger widths shown. Columns 1–2: test accuracy on the INR classification tasks, with the random-guessing baseline. Columns 3–4: test $R^{2}$ against ground-truth accuracies on the CNN generalization prediction task, with $R^{2}=0$ (mean-accuracy prediction) as baseline.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Base GMN model	Code setting	Duplication-compatible variant	Matrix-product variant
GMN (forward)	symmetry: permutation direction: forward	1. fan-in rescaled edges 2. mean aggregation 3. layer-wise mean readout	message $\overline{W}^{(\ell)}_{ij}A^{\mathrm{in}}_{\ell}h_{j}$ .
GMN (bidirectional)	symmetry: permutation direction: bidirectional
ScaleGMN (forward)	symmetry: scale direction: forward
ScaleGMN (bidirectional)	symmetry: scale direction: bidirectional		not defined

Appendix

Table 2 : The four base GMN architectures compared in our experiments, and their duplication-compatible and matrix-product variants. The forward and bidirectional ScaleGMN are given explicitly in Section C.3 , with the modifications specified. The forward and bidirectional GMN are defined analogously.

Model	Input	all three modifications	without fan-in rescaling	with sum aggregation	with sum readout ( 3 )
GMN (forward)	MLP	$1.8\times 10^{-7}\;\checkmark$	$4.3\times 10^{-3}\;\times$	$1.3\times 10^{-2}\;\times$	$1.8\times 10^{-1}\;\times$
GMN (bidirectional)	MLP	$2.4\times 10^{-7}\;\checkmark$	$9.9\times 10^{-3}\;\times$	$3.6\times 10^{-2}\;\times$	$1.2\times 10^{-1}\;\times$
ScaleGMN (forward)	MLP	$1.9\times 10^{-7}\;\checkmark$	$1.2\times 10^{-3}\;\times$	$6.7\times 10^{-1}\;\times$	$1.5\times 10^{-1}\;\times$
ScaleGMN (bidirectional)	MLP	$3.0\times 10^{-7}\;\checkmark$	$3.2\times 10^{-3}\;\times$	$5.2\times 10^{-1}\;\times$	$3.1\times 10^{-2}\;\times$
GMN (forward)	CNN	$1.6\times 10^{-7}\;\checkmark$	$7.4\times 10^{-4}\;\times$	$2.2\times 10^{-2}\;\times$	$1.4\times 10^{-2}\;\times$
GMN (bidirectional)	CNN	$1.6\times 10^{-7}\;\checkmark$	$9.6\times 10^{-4}\;\times$	$3.3\times 10^{-2}\;\times$	$2.6\times 10^{-2}\;\times$

Appendix

Table 3 : With all three modifications in place, the modified GMN become duplication-compatible; removing any one of them breaks this property. Entries are the maximum GMN output differences between an input network and its duplication widening ( 4 ), the maximum taken over the sampled input networks and over the coordinates of the model’s output.

Table 4 : The forward matrix-product model is compatible with the general block-wise widening; the other four families are subclasses of it and are hence also compatible. The bidirectional matrix-product model is compatible only with block-wise widenings that satisfy the column-balance condition of Proposition 3 (ii); the duplication widening and the doubly stochastic Kronecker widening are subclasses of this condition and are hence also compatible. The duplication-compatible model (forward or bidirectional) is compatible only with the duplication widening. The same pattern holds for both the GMN (a) and ScaleGMN families (b). Entries are the output difference between an input network and its widening, maximized over the sampled input networks and over the coordinates of the model’s output.

$n$	${\left\\|\theta_{n}\right\\|}_{V_{n},\mathrm{op}}$	Matrix-product GMN	Duplication-compatible GMN	Matrix-product GMN	Duplication-compatible GMN
		MLP weights		CNN weights
$16$	$0.2500$	$2.08\times 10^{-3}$	$4.99\times 10^{-3}$	$8.76\times 10^{-4}$	$6.68\times 10^{-4}$
$32$	$0.1768$	$1.04\times 10^{-3}$	$5.05\times 10^{-3}$	$4.38\times 10^{-4}$	$6.42\times 10^{-4}$
$64$	$0.1250$	$5.21\times 10^{-4}$	$5.08\times 10^{-3}$	$2.19\times 10^{-4}$	$6.30\times 10^{-4}$
$128$	$0.0884$	$2.60\times 10^{-4}$	$5.09\times 10^{-3}$	$1.10\times 10^{-4}$	$6.23\times 10^{-4}$
$256$	$0.0625$	$1.30\times 10^{-4}$	$5.10\times 10^{-3}$	—	—

Appendix

Table 5 : The matrix-product GMN is continuous in the normalized spectral norm, whereas the duplication-compatible GMN is not. Entries are

\left|f(\theta_{n})-f(0)\right|

on the Hadamard family, which should converge to

0

n\to\infty

for continuous models.

Panel of Figure 3 — Figure 3 : Size generalization on wider input networks generated with duplication widening ( 4 ). Both duplication-compatible and matrix-product GMNs generalize in size perfectly, with flat error curves across all test widths, while the baseline GMNs, which are not duplication-compatible, degrade sharply on wider input networks.

Panel of Figure 4 — Figure 4 : Size generalization on wider input networks generated with block-wise widening ( 6 ). Only the forward matrix-product GMNs/ScaleGMNs generalize in size perfectly, with flat error curves across all test widths, while all other models, which are not compatible with block-wise widenings, degrade sharply on wider input networks.

Dataset	Param.	$24$	$32$	$48$	$64$	$80$	$96$	$128$	$192$	$256$	$384$	$512$	$768$	$1024$
MNIST	SP	$30.2/0.7$	$31.9/0.3$	$34.5/0.2$	$34.8/0.2$	$35.4/0.4$	$35.9/0.5$	$35.0/0.4$	$32.6/0.7$	$33.4/1.1$	$31.1/1.1$	$31.7/1.0$	$29.3/1.0$	$29.5/3.2$
MNIST	$\mu$ P	$30.2/0.3$	$31.3/0.1$	$32.2/0.3$	$32.8/0.0$	$33.0/0.0$	$33.0/0.3$	$33.2/0.0$	$33.1/0.1$	$33.0/0.2$	$32.8/0.0$	$32.5/0.0$	$32.8/0.0$	$32.5/0.0$
FMNIST	SP	—	$25.5/5.9$	$27.3/2.7$	$27.7/2.8$	$27.6/2.3$	$27.9/2.2$	$28.0/3.9$	$27.1/4.3$	$27.0/5.4$	$25.6/7.5$	$25.1/10.9$	$24.1/13.6$	$23.3/19.5$
FMNIST	$\mu$ P	—	$25.5/4.8$	$26.1/4.4$	$26.6/3.4$	$26.9/3.0$	$27.2/2.3$	$27.3/1.8$	$27.3/2.4$	$27.4/1.9$	$27.5/1.5$	$27.4/1.7$	$27.4/2.2$	$27.3/2.7$

Appendix

Table 6 : Reconstruction quality of the fitted INRs at every width: peak signal-to-noise ratio (PSNR, in dB; higher is better) and failure rate (

\mathrm{MSE}>0.01

, in %). Under

\mu

P, quality generally stays flat across all widths; under SP it degrades past its peak, most severely on FMNIST.

Dataset	Param.	$32$	$48$	$64$	$96$	$128$	$192$	$256$	$384$	$512$
CIFAR-10	SP	$0.942/{+}0.031$	$0.889/{+}0.063$	$0.859/{+}0.079$	$0.818/{+}0.106$	$0.792/{+}0.112$	$0.728/{+}0.128$	$0.726/{+}0.136$	$0.654/{+}0.161$	$0.678/{+}0.154$
CIFAR-10	$\mu$ P	$0.958/{+}0.022$	$0.921/{+}0.033$	$0.884/{+}0.042$	$0.892/{+}0.054$	$0.880/{+}0.053$	$0.863/{+}0.062$	$0.863/{+}0.060$	$0.869/{+}0.063$	$0.864/{+}0.062$
SVHN	SP	$0.928/{+}0.000$	$0.876/{+}0.048$	$0.846/{+}0.079$	$0.799/{+}0.113$	$0.761/{+}0.130$	$0.715/{+}0.159$	$0.704/{+}0.174$	$0.652/{+}0.224$	$0.700/{+}0.197$
SVHN	$\mu$ P	$0.948/{+}0.000$	$0.911/{+}0.000$	$0.876/{+}0.000$	$0.880/{+}0.009$	$0.864/{+}0.000$	$0.850/{+}0.013$	$0.852/{+}0.011$	$0.855/{+}0.024$	$0.847/{+}0.002$

Appendix

Table 7 : Drift of the prediction target with width on the two CNN zoos, relative to the training width

c=16

. Each cell gives the fraction of that width’s test accuracies falling inside the central

98\%

range of the

c=16

accuracies, and the shift of the median accuracy (both are

1.000

and

+0.000

c=16

by definition).

Panel of Figure 5 — Figure 5 : Size generalization on wider input networks trained independently from random initialization, under either $\mu$ P or SP, for the two architectures not shown in the main text ( Figure 2 ).

Panel of Figure 6 — Figure 6 : The four evaluation criteria for accuracy prediction on the CIFAR-10 CNN zoo, one row of panels per setting.

Panel of Figure 7 — Figure 7 : The four evaluation criteria for accuracy prediction on the SVHN CNN zoo, one row of panels per setting.

Figure 8 : Hyperparameter transfer under

\mu

P, Figure 1 of Yang et al. [2021] . Training loss against learning rate for Transformers of width ranging over

128,\dots,8192

, trained with Adam.

Explore similar work

Train Small, Deploy Large: Zero-Shot GNN Transfer Through Geometric Renormalization
Jul 30, 2026Robert Jankowski, Pedro Almagro-Blanco, Marián Boguñá +2Graph Neural Networks Renormalization Group
TU Delft · University of Seville · University of Barcelona +2
Quasi-Equivariant Metanetworks
Apr 26, 2026Viet-Hoang Tran, An Nguyen, Benoît Guérand +2Equivariant Neural Networks Symmetry
Department of Mathematics National University of Singapore
Architecture Generalization with MetaNCA
Jul 8, 2026Meet Barot, Daniel Berenberg, Sina KhajehabdollahiNeural Cellular Automata Brain-Inspired Synergistic Framework
Mythos Scientific, New York, USA · Independent Scholar

Jul 30, 2026, cs.LGJ/K move · Enter open · S save

Train Small, Deploy Large: Zero-Shot GNN Transfer Through Geometric Renormalization

Robert Jankowski, Pedro Almagro-Blanco, Marián Boguñá, Melanie Weber+1

TU Delft · University of Seville · University of Barcelona+2

Graph Neural Networks Renormalization Group

Open paper AskPDF arXiv

Graph neural networks (GNNs) can operate on large graphs but become infrastructure-sensitive at the scale of millions of nodes and typically require scalable training techniques for even larger graphs. This raises a central question: when can a model trained on a smaller, scaled-down replica of a graph be deployed on the full-resolution graph without retraining? We introduce a zero-shot transfer protocol in which a GNN is trained on a graph coarse-grained by geometric renormalization (GR), and the resulting weights are transferred directly to the original network. Across synthetic and real-world networks, training on GR scaled-down replicas preserves much of the original-scale predictive performance while significantly reducing training cost. We further find that learned representations and predictive trajectories remain aligned across scales. These findings suggest that structural similarity may be more important than network size in determining GNN transferability, opening a path toward scale-equivariant graph architectures.

Abstract

Explore similar work

Train Small, Deploy Large: Zero-Shot GNN Transfer Through Geometric Renormalization

Quasi-Equivariant Metanetworks

Architecture Generalization with MetaNCA