Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks

Tilen Cadez; Sanghoon Lee; Kyoung-Min Kim

cs.LGOct 1, 2026

Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks

Authors: Tilen Cadez, Sanghoon Lee, Kyoung-Min Kim

Organizations: Asia Pacific Center for Theoretical Physics, Pohang, Gyeongbuk, 37673, Republic of Korea · Department of Physics, Pohang University of Science and Technology, Pohang, Gyeongbuk 37673, Republic of Korea

Abstract

Kolmogorov-Arnold Networks (KANs) represent a compelling alternative to traditional Multi-Layer Perceptron (MLP)-based neural networks. By employing activation functions as learnable elements, KANs offer superior interpretability, making them suited for scientific domains. In this work, we investigate the neural scaling laws of KANs and the structural evolution of their learnable activation functions under dataset expansion. Specifically, we evaluate the scaling behavior of three KAN variants---BSRBF-KAN, Gottlieb-KAN, and Faster-KAN---across standard image classification benchmarks (MNIST and Fashion-MNIST) and a specialized scientific regression task (magnetic parameter estimation from domain images of moiré magnetic textures). Our results demonstrate that the test loss ${\cal L}$ exhibits a broken neural scaling law (BNSL) behavior as a function of the dataset size $N_D$ . After passing through a random-guess regime, the loss follows architecture- and task-dependent scaling behavior. The loss crosses from a faster- to a slower-scaling branch, ${\cal L}\propto N_D^{-α}$ and ${\cal L}\propto N_D^{-β}$ with $α>β$ for image classification tasks. The exponents $α$ and $β$ depend strongly on both the specific network architecture and the dataset-size regime, ranging from 0.4 to 1.5 and from 0.06 to 0.6, respectively. For the magnetic parameter-regression task, the loss follows a single scaling law with its exponent ranging from 1.28 to 2.59. Additionally, we provide a structural analysis of how activation functions refine their complexity as data volume increases, finding that dataset expansion drives a transition from simple linear-like approximations toward stable, interpretable symbolic forms. These findings provide a quantitative roadmap for the efficient application of KANs while managing the trade-off between model expressivity and computational overhead.

Figures & tables

Figure 1: KAN architecture and benchmark datasets. (a) Schematic of the KAN architecture. In contrast to conventional MLPs that employ fixed nodal activation functions, KANs use learnable univariate activation functions on the edges. Circular nodes represent summation stages, while the spline curves on the edges visualize the diverse functional forms that evolve during the training. (b) Benchmark datasets used for neural scaling analysis: (Top) sample images from the MNIST handwritten digit dataset [ 16 ] , (Middle) Fashion-MNIST dataset [ 30 ] , and (Bottom) simulated moiré magnetic textures [ 6 ] . The bottom panel is reproduced from Ref. 7 under the terms of the Creative Commons Attribution (CC-BY) license.

Figure 2: Cross-entropy loss on MNIST. (a)–(c) show the loss versus training dataset size $N_{D}$ for BSRBF-KAN, Faster-KAN, and Gottlieb-KAN, respectively. The horizontal black dotted line marks the uniform random-guess level for ten class classification, $\log_{10}\mathcal{L}_{\rm CE}\simeq 0.362$ . Different colors denote the numbers of hidden neurons, $n_{h}=4,8,\ldots,256$ , while solid curves show the corresponding smoothly broken power-law (SBPL) fits. All three models exhibit broken neural scaling law (BNSL) behavior, crossing from the small-data random-guess (RG) regime through faster-scaling (FS) and slower-scaling (SS) regimes.

Figure 3: SBPL parameters on MNIST. The fitted SBPL quantities extracted from Fig. 2 are shown as functions of number of hidden neurons $n_{h}$ : the faster-scaling (FS) exponent $\alpha$ , the slower-scaling (SS) exponent $\beta$ , and the crossover dataset size $N_{\times}$ in panels (a)–(c), respectively. The corresponding fitting uncertainties are listed in Supplementary Note 1 Table S3.

Figure 4: Cross-entropy loss on Fashion-MNIST. Panels (a)–(c) show the loss versus training dataset size $N_{D}$ for BSRBF-KAN, Faster-KAN, and Gottlieb-KAN, respectively. The horizontal black dotted line marks the uniform random-guess level. Markers denote the number of hidden neurons $n_{h}=4,8,\ldots,256$ , and solid curves show the corresponding SBPL fits. As on MNIST [Fig. 2 ], all three models exhibit a finite-window BNSL from the small-data RG regime through the FS and SS regimes.

Figure 5: SBPL parameter trends on Fashion-MNIST. Panels (a)–(c) summarize the fitted SBPL quantities from Fig. 4 : the faster-scaling (FS) exponent $\alpha$ , the slower-scaling (SS) exponent $\beta$ , and the crossover dataset size $N_{\times}$ on the number of hidden neurons $n_{h}$ . The corresponding fitting uncertainties are listed in Supplementary Note 1 Table S4.

Figure 6: Initial and trained Gottlieb-KAN edge functions on MNIST. The upper and lower blocks show the edge functions $\Phi(x)$ for $n_{h}=64$ [(a)–(l)] and $n_{h}=256$ [(m)–(x)], respectively. For each block, the first column shows the shared initial edge functions in blue, remaining columns show the trained functions in black at $N_{D}=1000$ , $10000$ , and $60000$ . The random seed is fixed across the three dataset sizes. From top to bottom, the rows correspond to the input-to-hidden, hidden-to-hidden, and hidden-to-output connections, respectively. The vertical-axis range differs across the trained-function columns. The insets show the normalized magnitude $|\Phi(x)|/\max_{x^{\prime}}|\Phi(x^{\prime})|$ of each edge functions $\Phi(x)$ .

Figure 7: Test MSE scaling for Hamiltonian parameter regression. Geometric mean of the test mean-squared-error (MSE) for regression of the Hamiltonian parameters $\theta,J,D$ of twisted magnets using the BSRBF-KAN, Faster-KAN, and Gottlieb-KAN. The black dotted line denotes the random-guess loss, corresponds to $\log_{10}\mathcal{L}_{\rm MSE}=-1.303$ . Solid lines indicate power-law fits, $\smash{\mathcal{L}_{\rm MSE}\sim N_{D}^{-\alpha}}$ . Different numbers of edge functions per neuron were used with $n_{h}=4,16,64,256$ .

Figure 8: Scaling exponents for Hamiltonian parameter regression. Scaling exponents $\alpha$ , extracted from Figure 7 , for the MSE test loss scaling in the regression of the Hamiltonian parameters $\theta$ (blue), $J$ (red), $D$ (green) of twisted magnets using BSRBF-KAN (squares), Faster-KAN (circles), and Gottlieb-KAN (diamonds). Model sizes range from $31$ , $481$ to $5,225,280$ trainable parameters. For comparison, scaling exponents obtained with MLPs (hexagons), taken from Ref. 7 , are also shown.

Figure 9: Evolution of trained edge functions with dataset size. Initial (blue) and trained (black) Gottlieb-KAN edge functions for the $\theta$ regression task. Columns show for input-to-hidden (two left panels), hidden-to-hidden (two middle panels), and hidden-to-output (two right panels) connections; rows from top to bottom show $N_{D}=224$ , $7168$ , $57344$ , repsectively. At initialization, edge functions within each layer collapse onto a single shape family differing mainly by an overall scale factor of either sign. This pattern persists after training, with changes occurring mainly in the distribution of scales.

Figure 10: Evolution of the effective rank with dataset size. The independence of the effective rank $r_{\rm eff}$ , which remains close to one, as a function of training dataset size for regression of the twisted magnet Hamiltonian parameters $\theta,J,D$ . The mean over 10 realizations is calculated for each parameter and dataset size and the shaded area marks the standard deviation.

Explore similar work

Apr 23, 2026cs.CE

Scale-Parameter Selection in Gaussian Kolmogorov-Arnold Networks

Kolmogorov--Arnold Networks (KANs) have recently attracted attention as edge-based neural architectures in which learnable univariate functions replace conventional fixed activation functions. A key source of flexibility in KANs is the choice of basis functions used to parameterize the learnable edge functions. In this context, Gaussian basis functions provide a simple and efficient alternative to splines. However, their performance depends strongly on the scale (shape) parameter

ε

, whose role has not been studied systematically. In this paper, we investigate how

ε

affects Gaussian KANs through first-layer feature geometry, conditioning, and approximation behavior. Our central observation is that scale selection is governed primarily by the first layer, since it is the only layer constructed directly on the input domain and any loss of distinguishability introduced there cannot be recovered by later layers. From this viewpoint, we analyze the first-layer feature matrix and identify a practical operating interval,

ε\in \left[\frac{1}{G-1},\frac{2}{G-1}\right],

where

G

denotes the number of Gaussian centers. We interpret this interval not as a universal optimality result, but as a stable and effective design rule, and validate it through brute-force sweeps over

ε

across function-approximation problems with different collocation densities, grid resolutions, network architectures, and input dimensions, as well as physics-informed problems. We further show that this range is useful for fixed-scale selection, variable-scale constructions, constrained training of

ε

, and efficient scale search using early training MSE. In this way, the paper positions scale selection as a practical design principle for Gaussian KANs rather than as an ad hoc hyperparameter choice.

Amir Noorizadegan, Sifan Wang

Department of Mathematics, Hong Kong Baptist University, Hong Kong SAR, China · Institution for Foundations of Data Science, Yale University, New Haven, CT 06520, USA

Kolmogorov-Arnold Networks Basis Functions

Ask PDF

Sep 7, 2026cs.DC

Scalability Analysis of Distributed Kolmogorov-Arnold Network Training on High-Performance Computing Systems

Kolmogorov-Arnold Networks (KANs) replace the fixed activation functions and linear weights of Multi-Layer Perceptrons (MLPs) with learnable univariate functions on network edges, offering improved interpretability and, in some settings, competitive parameter efficiency. While the approximation properties of KANs have received considerable attention, their behavior under distributed, multi-GPU training has not been systematically characterized. This paper presents an empirical scalability study of data-parallel KAN training on multi-node, multi-GPU high-performance computing (HPC) infrastructure, evaluated along four dimensions: strong scaling, weak scaling, communication overhead, and model-size scaling. Experiments were conducted on the FinisTerrae III supercomputer using up to 8 NVIDIA A100 GPUs across 4 nodes with PyTorch Distributed Data Parallel (DDP). KAN training reaches 74.7% parallel efficiency at 8 GPUs with a 5.97x speedup, consistent with conventional deep learning workloads. Weak scaling shows an initial single-to-multi-GPU throughput drop followed by strong stability. Communication overhead follows a non-monotonic pattern (1.3%-6.1%), driven primarily by All-Reduce algorithm selection and inter-node latency rather than KAN's edge-wise gradient structure. The parameter-to-memory ratio improves with model size even as training time scales unfavorably. These results indicate that operator-level and data-parallel optimizations for KAN are complementary. We provide deployment guidelines for GPU topology and model-size selection, and discuss the limitations of a synthetic-regression evaluation.

Guangneng Chen, David Garcia Selfa, Pablo Quesada Barriuso

University of Santiago de Compostela, 15782 Santiago de Compostela, Spain · Galicia Supercomputing Center (CESGA), 15705 Santiago de Compostela, Spain

Kolmogorov-Arnold Networks High-Performance Computing Scalability Sequential Scaling

Ask PDF

Aug 1, 2026cs.LG

SparseKAN: Compressing Kolmogorov--Arnold Networks Across Basis Functions, Neurons, and Bits

Kolmogorov--Arnold Networks (KANs) replace scalar edge weights with learnable univariate functions parameterized by multiple basis coefficients. This introduces a source of redundancy that conventional neural-network compression does not directly expose. We present \textbf{SparseKAN}, a unified approach that compresses KANs along three complementary axes: basis functions, neurons/channels, and numerical precision. SparseKAN equips the base branch, nonlinear basis branch, and individual basis terms with hierarchical learnable gates trained under a differentiable active-cost objective. The learned importance structure is subsequently hardened under explicit basis and width budgets, recovered in full or low precision, and physically compacted into smaller dense tensors rather than retained as sparse masks. Experiments on MNIST, CIFAR-10, and CIFAR-100 across spline, polynomial, RBF, wavelet, and convolutional KAN variants show that the structural axes compose predictably in cost. We also find strong basis-dependent differences in term importance: coefficient-based selection outperforms matched low-order truncation by up to 15.25 accuracy points in the evaluated Gram-polynomial settings. Eight-bit quantization is broadly robust, whereas 4-bit convolutional KANs require quantization-aware adaptation. Physical compaction removes up to 73.0% of parameters without accuracy loss on MNIST and reduces large-batch CUDA latency to as little as

0.51\times

dense execution. On a ZCU104 FPGA, the resulting sparse low-bit models achieve up to

23.63\times

lower inference latency, demonstrating that SparseKAN converts functional redundancy into measurable software and hardware efficiency. The SparseKAN implementation is available at https://github.com/OSU-STARLAB/SparseKAN.

Kazi Ahmed Asif Fuad, Lizhong Chen

Department of EECS Oregon State University Corvallis, OR 97331

Kolmogorov-Arnold Networks Compressed Model Neurons Bits

Ask PDF