Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks

Tilen Cadez; Sanghoon Lee; Kyoung-Min Kim

cs.LGOct 1, 2026

Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks

Authors: Tilen Cadez, Sanghoon Lee, Kyoung-Min Kim

Organizations: Asia Pacific Center for Theoretical Physics, Pohang, Gyeongbuk, 37673, Republic of Korea · Department of Physics, Pohang University of Science and Technology, Pohang, Gyeongbuk 37673, Republic of Korea

Abstract

Kolmogorov-Arnold Networks (KANs) represent a compelling alternative to traditional Multi-Layer Perceptron (MLP)-based neural networks. By employing activation functions as learnable elements, KANs offer superior interpretability, making them suited for scientific domains. In this work, we investigate the neural scaling laws of KANs and the structural evolution of their learnable activation functions under dataset expansion. Specifically, we evaluate the scaling behavior of three KAN variants---BSRBF-KAN, Gottlieb-KAN, and Faster-KAN---across standard image classification benchmarks (MNIST and Fashion-MNIST) and a specialized scientific regression task (magnetic parameter estimation from domain images of moiré magnetic textures). Our results demonstrate that the test loss ${\cal L}$ exhibits a broken neural scaling law (BNSL) behavior as a function of the dataset size $N_D$ . After passing through a random-guess regime, the loss follows architecture- and task-dependent scaling behavior. The loss crosses from a faster- to a slower-scaling branch, ${\cal L}\propto N_D^{-α}$ and ${\cal L}\propto N_D^{-β}$ with $α>β$ for image classification tasks. The exponents $α$ and $β$ depend strongly on both the specific network architecture and the dataset-size regime, ranging from 0.4 to 1.5 and from 0.06 to 0.6, respectively. For the magnetic parameter-regression task, the loss follows a single scaling law with its exponent ranging from 1.28 to 2.59. Additionally, we provide a structural analysis of how activation functions refine their complexity as data volume increases, finding that dataset expansion drives a transition from simple linear-like approximations toward stable, interpretable symbolic forms. These findings provide a quantitative roadmap for the efficient application of KANs while managing the trade-off between model expressivity and computational overhead.

Figures & tables

Figure 1: KAN architecture and benchmark datasets. (a) Schematic of the KAN architecture. In contrast to conventional MLPs that employ fixed nodal activation functions, KANs use learnable univariate activation functions on the edges. Circular nodes represent summation stages, while the spline curves on the edges visualize the diverse functional forms that evolve during the training. (b) Benchmark datasets used for neural scaling analysis: (Top) sample images from the MNIST handwritten digit dataset [ 16 ] , (Middle) Fashion-MNIST dataset [ 30 ] , and (Bottom) simulated moiré magnetic textures [ 6 ] . The bottom panel is reproduced from Ref. 7 under the terms of the Creative Commons Attribution (CC-BY) license.

Figure 2: Cross-entropy loss on MNIST. (a)–(c) show the loss versus training dataset size $N_{D}$ for BSRBF-KAN, Faster-KAN, and Gottlieb-KAN, respectively. The horizontal black dotted line marks the uniform random-guess level for ten class classification, $\log_{10}\mathcal{L}_{\rm CE}\simeq 0.362$ . Different colors denote the numbers of hidden neurons, $n_{h}=4,8,\ldots,256$ , while solid curves show the corresponding smoothly broken power-law (SBPL) fits. All three models exhibit broken neural scaling law (BNSL) behavior, crossing from the small-data random-guess (RG) regime through faster-scaling (FS) and slower-scaling (SS) regimes.

Figure 3: SBPL parameters on MNIST. The fitted SBPL quantities extracted from Fig. 2 are shown as functions of number of hidden neurons $n_{h}$ : the faster-scaling (FS) exponent $\alpha$ , the slower-scaling (SS) exponent $\beta$ , and the crossover dataset size $N_{\times}$ in panels (a)–(c), respectively. The corresponding fitting uncertainties are listed in Supplementary Note 1 Table S3.

Figure 4: Cross-entropy loss on Fashion-MNIST. Panels (a)–(c) show the loss versus training dataset size $N_{D}$ for BSRBF-KAN, Faster-KAN, and Gottlieb-KAN, respectively. The horizontal black dotted line marks the uniform random-guess level. Markers denote the number of hidden neurons $n_{h}=4,8,\ldots,256$ , and solid curves show the corresponding SBPL fits. As on MNIST [Fig. 2 ], all three models exhibit a finite-window BNSL from the small-data RG regime through the FS and SS regimes.

Figure 5: SBPL parameter trends on Fashion-MNIST. Panels (a)–(c) summarize the fitted SBPL quantities from Fig. 4 : the faster-scaling (FS) exponent $\alpha$ , the slower-scaling (SS) exponent $\beta$ , and the crossover dataset size $N_{\times}$ on the number of hidden neurons $n_{h}$ . The corresponding fitting uncertainties are listed in Supplementary Note 1 Table S4.

Figure 6: Initial and trained Gottlieb-KAN edge functions on MNIST. The upper and lower blocks show the edge functions $\Phi(x)$ for $n_{h}=64$ [(a)–(l)] and $n_{h}=256$ [(m)–(x)], respectively. For each block, the first column shows the shared initial edge functions in blue, remaining columns show the trained functions in black at $N_{D}=1000$ , $10000$ , and $60000$ . The random seed is fixed across the three dataset sizes. From top to bottom, the rows correspond to the input-to-hidden, hidden-to-hidden, and hidden-to-output connections, respectively. The vertical-axis range differs across the trained-function columns. The insets show the normalized magnitude $|\Phi(x)|/\max_{x^{\prime}}|\Phi(x^{\prime})|$ of each edge functions $\Phi(x)$ .

Figure 7: Test MSE scaling for Hamiltonian parameter regression. Geometric mean of the test mean-squared-error (MSE) for regression of the Hamiltonian parameters $\theta,J,D$ of twisted magnets using the BSRBF-KAN, Faster-KAN, and Gottlieb-KAN. The black dotted line denotes the random-guess loss, corresponds to $\log_{10}\mathcal{L}_{\rm MSE}=-1.303$ . Solid lines indicate power-law fits, $\smash{\mathcal{L}_{\rm MSE}\sim N_{D}^{-\alpha}}$ . Different numbers of edge functions per neuron were used with $n_{h}=4,16,64,256$ .

Figure 8: Scaling exponents for Hamiltonian parameter regression. Scaling exponents $\alpha$ , extracted from Figure 7 , for the MSE test loss scaling in the regression of the Hamiltonian parameters $\theta$ (blue), $J$ (red), $D$ (green) of twisted magnets using BSRBF-KAN (squares), Faster-KAN (circles), and Gottlieb-KAN (diamonds). Model sizes range from $31$ , $481$ to $5,225,280$ trainable parameters. For comparison, scaling exponents obtained with MLPs (hexagons), taken from Ref. 7 , are also shown.

Figure 9: Evolution of trained edge functions with dataset size. Initial (blue) and trained (black) Gottlieb-KAN edge functions for the $\theta$ regression task. Columns show for input-to-hidden (two left panels), hidden-to-hidden (two middle panels), and hidden-to-output (two right panels) connections; rows from top to bottom show $N_{D}=224$ , $7168$ , $57344$ , repsectively. At initialization, edge functions within each layer collapse onto a single shape family differing mainly by an overall scale factor of either sign. This pattern persists after training, with changes occurring mainly in the distribution of scales.

Figure 10: Evolution of the effective rank with dataset size. The independence of the effective rank $r_{\rm eff}$ , which remains close to one, as a function of training dataset size for regression of the twisted magnet Hamiltonian parameters $\theta,J,D$ . The mean over 10 realizations is calculated for each parameter and dataset size and the shaded area marks the standard deviation.

Explore similar work

Scale-Parameter Selection in Gaussian Kolmogorov-Arnold Networks
Apr 23, 2026Amir Noorizadegan, Sifan WangKolmogorov-Arnold Networks Basis Functions
Department of Mathematics, Hong Kong Baptist University, Hong Kong SAR, China · Institution for Foundations of Data Science, Yale University, New Haven, CT 06520, USA
Scalability Analysis of Distributed Kolmogorov-Arnold Network Training on High-Performance Computing Systems
Sep 7, 2026Guangneng Chen, David Garcia Selfa, Pablo Quesada BarriusoKolmogorov-Arnold Networks High-Performance Computing
University of Santiago de Compostela, 15782 Santiago de Compostela, Spain · Galicia Supercomputing Center (CESGA), 15705 Santiago de Compostela, Spain
SparseKAN: Compressing Kolmogorov--Arnold Networks Across Basis Functions, Neurons, and Bits
Aug 1, 2026Kazi Ahmed Asif Fuad, Lizhong ChenKolmogorov-Arnold Networks Compressed Model
Department of EECS Oregon State University Corvallis, OR 97331

Apr 23, 2026, cs.CEJ/K move · Enter open · S save

Scale-Parameter Selection in Gaussian Kolmogorov-Arnold Networks

Amir Noorizadegan, Sifan Wang

Department of Mathematics, Hong Kong Baptist University, Hong Kong SAR, China · Institution for Foundations of Data Science, Yale University, New Haven, CT 06520, USA

Kolmogorov-Arnold Networks Basis Functions

Open paper AskPDF arXiv

Kolmogorov--Arnold Networks (KANs) have recently attracted attention as edge-based neural architectures in which learnable univariate functions replace conventional fixed activation functions. A key source of flexibility in KANs is the choice of basis functions used to parameterize the learnable edge functions. In this context, Gaussian basis functions provide a simple and efficient alternative to splines. However, their performance depends strongly on the scale (shape) parameter

ε

, whose role has not been studied systematically. In this paper, we investigate how

ε

affects Gaussian KANs through first-layer feature geometry, conditioning, and approximation behavior. Our central observation is that scale selection is governed primarily by the first layer, since it is the only layer constructed directly on the input domain and any loss of distinguishability introduced there cannot be recovered by later layers. From this viewpoint, we analyze the first-layer feature matrix and identify a practical operating interval,

ε\in \left[\frac{1}{G-1},\frac{2}{G-1}\right],

where

G

denotes the number of Gaussian centers. We interpret this interval not as a universal optimality result, but as a stable and effective design rule, and validate it through brute-force sweeps over

ε

across function-approximation problems with different collocation densities, grid resolutions, network architectures, and input dimensions, as well as physics-informed problems. We further show that this range is useful for fixed-scale selection, variable-scale constructions, constrained training of

ε

, and efficient scale search using early training MSE. In this way, the paper positions scale selection as a practical design principle for Gaussian KANs rather than as an ad hoc hyperparameter choice.

Abstract

Explore similar work

Scale-Parameter Selection in Gaussian Kolmogorov-Arnold Networks

Scalability Analysis of Distributed Kolmogorov-Arnold Network Training on High-Performance Computing Systems

SparseKAN: Compressing Kolmogorov--Arnold Networks Across Basis Functions, Neurons, and Bits