A Graph Meta-Network for Learning on Kolmogorov-Arnold Networks

Guy Bar-Shalom; Ami Tavory; Itay Evron; Maya Bechler-Speicher; Ido Guy; Haggai Maron

cs.LGFeb 18, 2026

A Graph Meta-Network for Learning on Kolmogorov-Arnold Networks

Authors: Guy Bar-Shalom, Ami Tavory, Itay Evron, Maya Bechler-Speicher, Ido Guy, Haggai Maron

Organizations: Technion, Meta · Meta · Meta, Ben-Gurion University of the Negev · Technion

Abstract

Weight-space models learn directly from the parameters of neural networks, enabling tasks such as predicting their accuracy on new datasets. Naive methods -- like applying MLPs to flattened parameters -- perform poorly, making the design of better weight-space architectures a central challenge. While prior work leveraged permutation symmetries in standard networks to guide such designs, no analogous analysis or tailored architecture yet exists for Kolmogorov-Arnold Networks (KANs). In this work, we show that KANs share the same permutation symmetries as MLPs, and propose the KAN-graph, a graph representation of their computation. Building on this, we develop WS-KAN, the first weight-space architecture that learns on KANs, which naturally accounts for their symmetry. We analyze WS-KAN's expressive power, showing it can replicate an input KAN's forward pass - a standard approach for assessing expressiveness in weight-space architectures. We construct a comprehensive ``zoo'' of trained KANs spanning diverse tasks, which we use as benchmarks to empirically evaluate WS-KAN. Across all tasks, WS-KAN consistently outperforms structure-agnostic baselines, often by a substantial margin. Our code is available at https://github.com/BarSGuy/KAN-Graph-Metanetwork.

Figures & tables

Figure 1: Constructing the KAN-graph for a given Kolmogorov-Arnold Network (KAN).

Figure 2: Hidden neuron permutation symmetries in KANs.

Figure 3

Figure 3: PE.

Method	MNIST	F-MNIST	CIFAR-10
MLP	34.1 $\pm$ 0.1	41.3 $\pm$ 0.3	16.8 $\pm$ 0.1
MLP + Aug.	62.7 $\pm$ 1.1	63.0 $\pm$ 0.3	28.2 $\pm$ 0.7
MLP + Align.	81.0 $\pm$ 0.1	73.6 $\pm$ 0.2	30.0 $\pm$ 0.2
DMC	73.4 $\pm$ 3.0	73.1 $\pm$ 1.0	33.0 $\pm$ 0.7
DS (Ours)	59.1 $\pm$ 3.3	65.9 $\pm$ 0.8	23.2 $\pm$ 3.9
SetTrans (Ours)	87.5 $\pm$ 0.8	80.2 $\pm$ 0.1	34.3 $\pm$ 0.7

Table 1: INR classification accuracy.

Dataset	$h=32$	$h=48$	$h=64$	$h=80$	$h=96$
MNIST	94.3 $\pm$ 0.5	91.4 $\pm$ 0.5	81.0 $\pm$ 3.2	67.0 $\pm$ 4.3	57.1 $\pm$ 6.1
F-MNIST	84.6 $\pm$ 0.6	84.6 $\pm$ 0.6	84.3 $\pm$ 0.7	83.3 $\pm$ 0.8	82.2 $\pm$ 0.7

Table 2: OOD generalization for INR classification. Test accuracies of WS-KAN on the INR classification task when trained on KANs with hidden width

h=32

and evaluated on wider, previously unseen architectures (

h\in\{48,64,80,96\}

). All architectures follow the topology

[2\to h\to h\to 10]

. The blue column denotes the in-distribution setting , while red columns denote out-of-distribution setting .

Figure 9

	MSE (lower is better; $\downarrow$ ) [ $\times 10^{3}$ ]			$\textbf{R}^{2}$ (higher is better; $\uparrow$ ) [ $\times 10^{2}$ ]
Method	MNIST	F-MNIST	K-MNIST	MNIST	F-MNIST	K-MNIST
MLP	14.58 $\pm$ 0.02	11.55 $\pm$ 0.12	6.68 $\pm$ 0.16	76.99 $\pm$ 0.04	69.59 $\pm$ 0.33	80.13 $\pm$ 0.49
MLP + Aug.	10.41 $\pm$ 0.24	8.86 $\pm$ 1.47	5.33 $\pm$ 0.19	83.56 $\pm$ 0.38	76.66 $\pm$ 3.88	84.15 $\pm$ 0.55
MLP + Align.	5.26 $\pm$ 0.09	6.32 $\pm$ 0.15	3.33 $\pm$ 0.10	91.70 $\pm$ 0.15	83.37 $\pm$ 0.41	90.11 $\pm$ 0.31
DMC	7.00 $\pm$ 0.06	6.46 $\pm$ 0.32	3.27 $\pm$ 0.05	88.95 $\pm$ 0.09	82.99 $\pm$ 0.85	90.29 $\pm$ 0.14
DS (Ours)	3.29 $\pm$ 0.12	3.90 $\pm$ 0.04	2.00 $\pm$ 0.11	94.81 $\pm$ 0.18	89.73 $\pm$ 0.11	94.07 $\pm$ 0.32

Table 3: Accuracy prediction . Comparison of MSE and

R^{2}

across datasets.

	Accuracy ( $\uparrow$ , %)			ROC-AUC ( $\uparrow$ , %)
Method	MNIST	F-MNIST	K-MNIST	MNIST	F-MNIST	K-MNIST
MLP	93.10 $\pm$ $<$ $0.01$	96.65 $\pm$ $<$ $0.01$	91.39 $\pm$ $<$ $0.01$	87.12 $\pm$ 0.04	84.92 $\pm$ 0.38	75.32 $\pm$ 0.02
MLP + Aug.	93.29 $\pm$ $<$ $0.01$	96.65 $\pm$ $<$ $0.01$	91.39 $\pm$ $<$ $0.01$	91.36 $\pm$ 0.12	86.21 $\pm$ 0.09	74.89 $\pm$ 0.01
MLP + Align.	93.57 $\pm$ 0.01	96.64 $\pm$ $<$ $0.01$	91.52 $\pm$ 0.01	93.00 $\pm$ 0.03	91.66 $\pm$ 0.10	82.76 $\pm$ 0.05
DMC	93.07 $\pm$ $<$ $0.01$	96.59 $\pm$ $<$ $0.01$	91.39 $\pm$ $<$ $0.01$	84.27 $\pm$ 0.06	84.39 $\pm$ 0.01	75.06 $\pm$ 0.02
DS (Ours)	94.34 $\pm$ 0.02	96.90 $\pm$ 0.09	94.34 $\pm$ 0.02	95.45 $\pm$ 0.05	95.81 $\pm$ 0.04	95.45 $\pm$ 0.05

Table 4: Pruning mask prediction.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Example of a KAN-based INR applied to the synthetic 2D sine wave dataset. The left panel shows the ground truth, and the right panel shows the reconstructed result.

Panel of Figure 7 — Figure 7: Reconstructions from INR on CIFAR-10, Fashion-MNIST, and MNIST. All PSNRs are more than 40.

Figure 8: Results on the synthetic 2D sine wave INR dataset.

Panel of Figure 9 — Figure 9: KANs test accuracies. Scatter plots of KAN accuracies on the test set (Y-axis) as a function of the number of noisy labels (X-axis).

Figure 10: Weight-space training for accuracy prediction . We train KANs under varying noise levels, then fit a WS model (e.g., WS-KAN ) to predict their accuracy from weights.

Parameter	Values
Embedding size	$\{128,32\}$
Number of epochs	$100$
Learning rate	$\{0.001,0.005,0.0001\}$
Dropout	$0.2$
Number of layers	$\{4,1\}$
Batch size	$32$

Appendix

Table 5: Hyperparameter configuration used in the experiments for accuracy prediction.

Figure 11: Pipeline for training a WS model for the task of pruning mask prediction.

Panel of Figure 12 — Figure 12: Downstream pruning performance across methods over F-MNIST . We report: (i) Test accuracy : the downstream accuracy of pruned networks, averaged over non-overlapping bins of 20%, to highlight the relative effectiveness of pruning strategies under varying noise levels – Figure 12(a) ; (ii) Kept weights : the percentage of weights retained after pruning, averaged over the same bins as in (i) – Figure 12(b) .

Panel of Figure 13 — Figure 13: Downstream pruning performance across methods over K-MNIST . We report: (i) Test accuracy : the downstream accuracy of pruned networks, averaged over non-overlapping bins of 20%, to highlight the relative effectiveness of pruning strategies under varying noise levels – Figure 13(a) ; (ii) Kept weights : the percentage of weights retained after pruning, averaged over the same bins as in (i) – Figure 13(b) .

Models	MNIST	F-MNIST	CIFAR10
WS-KAN – no PE	$\mathbf{94.3\pm 0.4}$	$84.5\pm 0.1$	$\mathbf{46.1\pm 1.2}$
WS-KAN – no BIDIR	$93.8\pm 0.2$	$83.4\pm 0.1$	$10.0\pm{\scriptstyle<}0.1$
WS-KAN	$\mathbf{94.3\pm 0.5}$	$\mathbf{84.6\pm 0.6}$	$42.2\pm 0.8$

Appendix

Table 6: Ablation study – INR classification accuracy.

Model	MNIST (MSE)	F-MNIST (MSE)	K-MNIST (MSE)
WS-KAN – no PE	$\mathbf{2.53\pm 0.22}$	$\mathbf{2.44\pm 0.50}$	$\mathbf{1.10\pm 0.17}$
WS-KAN – no BIDIR	$4.03\pm 0.62$	$3.64\pm 0.12$	$1.50\pm 0.08$
WS-KAN	$3.29\pm 0.17$	$2.94\pm 0.13$	$1.45\pm 0.08$

Appendix

Table 7: Ablation study – Accuracy prediction (MSE) [

\times 10^{3}

Model	MNIST ( $R^{2}$ )	F-MNIST ( $R^{2}$ )	K-MNIST ( $R^{2}$ )
WS-KAN – no PE	$\mathbf{96.00\pm 0.35}$	$\mathbf{93.57\pm 1.32}$	$\mathbf{96.62\pm 0.51}$
WS-KAN – no BIDIR	$93.64\pm 0.98$	$90.40\pm 0.31$	$95.54\pm 0.23$
WS-KAN	$94.81\pm 0.27$	$92.27\pm 0.35$	$95.69\pm 0.24$

Appendix

Table 8: Ablation study – Accuracy prediction (

R^{2}

Model	MNIST (Accuracy)	F-MNIST (Accuracy)	K-MNIST (Accuracy)
WS-KAN – no PE	$97.77\pm 0.28$	$98.68\pm 0.04$	$96.28\pm 0.45$
WS-KAN – no BIDIR	$96.74\pm 0.27$	$98.27\pm 0.04$	$95.39\pm 0.14$
WS-KAN	$\mathbf{97.93\pm 0.19}$	$\mathbf{98.93\pm 0.05}$	$\mathbf{97.72\pm 0.14}$

Appendix

Table 9: Ablation study – Pruning mask prediction accuracy.

Model	MNIST (AUC)	F-MNIST (AUC)	K-MNIST (AUC)
WS-KAN – no PE	$99.51\pm 0.09$	$99.59\pm 0.02$	$98.85\pm 0.21$
WS-KAN – no BIDIR	$98.38\pm 0.20$	$98.99\pm 0.01$	$97.11\pm 0.37$
WS-KAN	$\mathbf{99.54\pm 0.01}$	$\mathbf{99.72\pm 0.02}$	$\mathbf{99.46\pm 0.09}$

Appendix

Table 10: Ablation study – Pruning mask prediction AUC.

Dataset	MNIST	F-MNIST	CIFAR-10
Train time per epoch (s)	$10.12\pm 0.06$	$10.18\pm 0.05$	$10.88\pm 0.10$
Test time per epoch (s)	$2.84\pm 0.12$	$2.81\pm 0.06$	$3.05\pm 0.08$

Appendix

Table 11: Running times measured (in seconds) for all experiments conducted on the INR classification task.

Explore similar work

Kolmogorov--Arnold Networks for Small Language Models
Jul 17, 2026Felippe Alves, Renato VicenteSmall Language Models Neural Network Interpretability
TELUS Digital Research Hub – CIAAM Institute of Mathematics, Statistics and Computer Science University of São Paulo
SparseKAN: Compressing Kolmogorov--Arnold Networks Across Basis Functions, Neurons, and Bits
Aug 1, 2026Kazi Ahmed Asif Fuad, Lizhong ChenKolmogorov-Arnold Networks Neural Network Quantization
Department of EECS Oregon State University Corvallis, OR 97331
KANLib -- A Modular, Extensible and Fast Kolmogorov-Arnold Network Implementation
Jun 16, 2026Julian Hoever, Gregor SchieleKolmogorov-Arnold Networks Modular Neural Networks
Intelligent Embedded Systems University of Duisburg-Essen Duisburg, Germany

Jul 17, 2026J/K move · Enter open · S save

Kolmogorov--Arnold Networks for Small Language Models

Felippe Alves, Renato Vicente

TELUS Digital Research Hub – CIAAM Institute of Mathematics, Statistics and Computer Science University of São Paulo

Small Language Models Neural Network Interpretability Kolmogorov-Arnold Networks Feedforward Neural Networks

Open paper AskPDF arXiv

Kolmogorov--Arnold Networks (KANs) replace fixed node activations with learned one-dimensional edge functions, offering an explicit interface for interpretation and a possible alternative to transformer feed-forward networks. We test these claims separately. In a six-layer, 10M-parameter B-spline KAN, we reconstruct all 884,736 feed-forward edges: 87.8% exceed (NLS>0.1) and 0.4% are inactive. Pruning the lowest-activity 20--25% causes negligible loss increase, although structured MLP neuron pruning tolerates comparable sparsity. The audit replicates on BabyLM, but grid-size sweeps show that near-total fPCA compression and high closed-form-fit coverage are properties of the low-capacity grid-2 basis, not universal KAN behavior. For replacement, we evaluate MLP, SwiGLU, grouped Chebyshev, and rational GR-KAN networks on BabyLM. The KAN-family and gated variants improve validation loss over the GELU MLP, but this ordering does not transfer to standardized benchmarks: across ten seeds and 59,875 BLiMP pairs, accuracies span 62.4--63.1%, EWoK remains at chance, and a (+0.7)-point GR-KAN effect on BLiMP reverses on the supplement. Larger tests are also cautionary: parameter-matched MLPEdge underperforms the MLP on Wikitext-103, and 286M-parameter GR-KAN remains below a SwiGLU ClimbMix baseline after stabilization. Thus, small-basis KANs provide a practical, corpus-transferable interface for auditing learned scalar transformations, but the tested replacements show no consistent benchmark, quality, or latency advantage over strong MLP baselines.

Abstract

Explore similar work

Kolmogorov--Arnold Networks for Small Language Models

SparseKAN: Compressing Kolmogorov--Arnold Networks Across Basis Functions, Neurons, and Bits

KANLib -- A Modular, Extensible and Fast Kolmogorov-Arnold Network Implementation