Kolmogorov-Arnold Networks: Curse of Dimensionality and Structural Bias

Yunhao Ni; Yuxin Guo; Jie Luo; Wenjun Wu; Lei Huang

cs.LGOct 15, 2026

Kolmogorov-Arnold Networks: Curse of Dimensionality and Structural Bias

Authors: Yunhao Ni, Yuxin Guo, Jie Luo, Wenjun Wu, Lei Huang

Organizations: State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

Abstract

Kolmogorov-Arnold networks (KANs) have a dimension-independent approximation exponent when the target admits a sufficiently smooth representation by univariate edge functions. We ask whether this favorable scaling belongs to the architecture or to the assumed target class. For arbitrary-depth standard spline-KANs, we prove a worst-case lower bound on isotropic Sobolev balls: below a dimension-independent accuracy threshold, the required number of spline coefficients grows at least exponentially with input dimension. Existing composition-matched MLP results, by contrast, show that favorable exponents on low-arity targets are not exclusive to KANs. We also revisit empirical comparisons. On four scaling-law targets, spline-KAN approximants constructed from known decompositions transfer exactly to MLPs with the cubic ReLU activation $σ(x)=(\max\{0,x\})^3$ , although their full dense parameter counts are larger. In controlled ridge-profile scans, KANs have a strong advantage when each nonlinear component depends on one input coordinate. When a component couples multiple input coordinates, this advantage may narrow sharply. At tight budgets, interacting targets often favor MLPs. In the random-label test on MNIST, ReLU-MLPs fit the randomized training labels more efficiently than spline-only KANs. Together, these results locate favorable scaling in target structure and a compatible inductive bias, not a general KAN-specific escape from the curse of dimensionality.

Figures & tables

Figure 1: Constructive scaling on four targets from Liu et al. (2025b) . Blue: constructed degree-three spline-KAN test RMSE versus spline-coefficient count. Purple: transferred ReLU 3 MLP test RMSE versus dense parameter count, at the validation-selected checkpoint (step zero or up to 200 Adam steps). At step zero the MLP represents the same function as its KAN source. All panels use

G\in\{8,16,32,64,128\}

Figure 2: Training-probe-selected test-MSE ratios on square-ridge targets with

d=16

and

R\in\{4,16,64\}

. Each family is selected independently within budget

P

. MLP candidates include four activations. The ratio

\rho>1

favors KAN, and

s

is the number of active coordinates per ridge. The full nine-panel scan is in Appendix C.3 .

Labels	Model	$P\leq 5\mathrm{k}$	$P\leq 20\mathrm{k}$	$P\leq 100\mathrm{k}$	$P\leq 500\mathrm{k}$
Standard	ReLU-MLP	87.5	94.9	98.0	98.6
Standard	Spline-only KAN	54.8	89.6	96.0	97.3
Random	ReLU-MLP	11.7	18.2	47.6	100.0
Random	Spline-only KAN	12.3	13.4	16.9	19.4

Table 1: MNIST accuracies (%) with

N_{\mathrm{train}}=60{,}000

within selected trainable-parameter budgets. Standard labels report test accuracy and random labels report final training accuracy. Each entry is the best architecture median over three seeds after selecting optimizer and learning rate by final training accuracy. Bold marks the higher median within each label condition and budget.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Constructive scaling on four targets from Liu et al. (2025b) . Blue: constructed cubic spline-KAN test RMSE versus spline-coefficient count. Purple: transferred ReLU 3 MLP test RMSE versus dense parameter count, at the validation-selected checkpoint (step zero or up to 200 Adam steps). At step zero the MLP represents the same function as its KAN source. All panels use

G\in\{8,16,32,64,128\}

Item	Setting
Target grid	$d,R\in\{4,16,64\}$ , and $s\in\{1,2,4,\ldots,d\}$
Target profiles	$\varphi(t)=t^{2}$ , $\sin(2\pi t)$ , $\tanh(t)$ , or $\operatorname{SiLU}(t)=t/(1+e^{-t})$
Hidden depth	$L\in\{2,4,6\}$ for both families
MLP	Width $w\in\{1,4,16,64,256\}$ ; ReLU, tanh, SiLU, or sine activation
Spline-only KAN	Width $w\in\{1,4,16,64\}$ ; $k=3$ ; $G\in\{3,7,15,31\}$ ; zero edge-wise base branch
Optimization	MSE; Adam; zero weight decay; 5,000 online-sampling steps; batch size 4,096; 500-step linear warmup followed by cosine decay to $1\%$ of the base rate

Appendix

Table 2: Controlled ridge-profile scan.

Figure 4: Complete square-ridge test-MSE ratios for

d,R\in\{4,16,64\}

, under the same training-probe selection rule as Figure 2 . The middle row is shown in the main text.

Figure 5: Sine-ridge test-MSE ratios under the same three-seed scan and parameter-budget selection as Figure 2 .

Figure 6: SiLU-ridge test-MSE ratios under the same scan and selection rule.

Figure 7: Tanh-ridge test-MSE ratios under the same scan and selection rule.

Figure 8: Sine-ridge test-error ratios when the MLP envelope is restricted to ReLU, tanh, or SiLU. Plotting and selection conventions follow Figure 5 for the

d\in\{4,16\}

subset. Removing sine activation expands the KAN-favored region, while denser supports still narrow that advantage.

Figure 9: Sine-ridge test-error ratios for a fixed ReLU-MLP comparator. Increasing support

s

again shifts the comparison toward MLP with a ReLU comparator.

Figure 10: Training-probe MSE ratios for the best-activation MLP comparison. The panels cover

d\in\{4,16\}

with the same selection rule as Figure 5 . The support-dependent pattern matches the independent-test ratios.

Figure 11: Absolute MSE of the models selected for the best-activation sine-ridge envelope. Solid curves report independent-test MSE and dashed curves report training-probe MSE. The dotted horizontal line marks MSE one, the zero-predictor baseline after target standardization.

Item	Setting
Hidden depth	$L\in\{2,4,6,8\}$
Hidden width	MLP: $w\in\{4,8,16,32,64,128,256,512\}$ ; KAN: $w\in\{1,2,4,8,16,32\}$
MLP	ReLU activation; Kaiming-uniform weights; zero biases
Spline-only KAN	$G=3$ , $k=3$ ; zero edge-wise base branch; spline coefficients trainable and spline scale fixed
Optimization	Cross-entropy, batch size 256, 200 epochs, no weight decay; SGD with momentum $0.9$ and learning rate in $\{0.1,0.03,0.01\}$ , or Adam with learning rate in $\{0.01,0.003,0.001\}$
Aggregation	For each architecture and seed, retain the optimizer/rate with highest final training accuracy; report medians and interquartile ranges over three seeds

Appendix

Table 3: MNIST classification and random-label fitting hyperparameters.

Figure 12: Full MNIST accuracy frontiers with

N_{\mathrm{train}}=60{,}000

. The left panel reports standard-label test accuracy and the right panel reports random-label final training accuracy. Each point is a three-seed architecture median after optimizer and learning-rate selection by final training accuracy. The curves show the best available value within each parameter budget.

Figure 13: Depth–width scan for normal-label testing (top) and random-label fitting (bottom). Cell values are median accuracies in percent. Width grids differ between MLP and KAN so that the scan covers overlapping parameter budgets. The plot title’s $N=60{,}000$ denotes the number of training examples. The random-label fitting gap persists across the tested depths.

Figure 14: Small-sample MNIST comparison with

N_{\mathrm{train}}=6{,}000

. The normal-label panel reports median test accuracy and the random-label panel reports median training accuracy over three seeds. The display is truncated just beyond the largest scanned KAN budget so that the visible range is covered by both families.

Curse of Dimensionality

See all 9 papers

Kolmogorov--Arnold Networks for Small Language Models
Jul 17, 2026Felippe Alves, Renato VicenteSmall Language Models Neural Network Interpretability
TELUS Digital Research Hub – CIAAM Institute of Mathematics, Statistics and Computer Science University of São Paulo
Multilevel Training for Kolmogorov Arnold Networks
Mar 5, 2026Ben S. Southworth, Jonas A. Actor, Graham Harper +1Kolmogorov-Arnold Networks Neural Network Optimization
Theoretical Division, Los Alamos National Laboratory, USA. · Center for Computing Research, Sandia National Laboratories, USA.
Necessary and sufficient conditions for universality of Kolmogorov-Arnold networks
Apr 26, 2026Vugar IsmailovKolmogorov-Arnold Networks Universal Approximation

Jul 17, 2026J/K move · Enter open · S save

Kolmogorov--Arnold Networks for Small Language Models

Felippe Alves, Renato Vicente

TELUS Digital Research Hub – CIAAM Institute of Mathematics, Statistics and Computer Science University of São Paulo

Small Language Models Neural Network Interpretability Kolmogorov-Arnold Networks Feedforward Neural Networks

Open paper AskPDF arXiv

Kolmogorov--Arnold Networks (KANs) replace fixed node activations with learned one-dimensional edge functions, offering an explicit interface for interpretation and a possible alternative to transformer feed-forward networks. We test these claims separately. In a six-layer, 10M-parameter B-spline KAN, we reconstruct all 884,736 feed-forward edges: 87.8% exceed (NLS>0.1) and 0.4% are inactive. Pruning the lowest-activity 20--25% causes negligible loss increase, although structured MLP neuron pruning tolerates comparable sparsity. The audit replicates on BabyLM, but grid-size sweeps show that near-total fPCA compression and high closed-form-fit coverage are properties of the low-capacity grid-2 basis, not universal KAN behavior. For replacement, we evaluate MLP, SwiGLU, grouped Chebyshev, and rational GR-KAN networks on BabyLM. The KAN-family and gated variants improve validation loss over the GELU MLP, but this ordering does not transfer to standardized benchmarks: across ten seeds and 59,875 BLiMP pairs, accuracies span 62.4--63.1%, EWoK remains at chance, and a (+0.7)-point GR-KAN effect on BLiMP reverses on the supplement. Larger tests are also cautionary: parameter-matched MLPEdge underperforms the MLP on Wikitext-103, and 286M-parameter GR-KAN remains below a SwiGLU ClimbMix baseline after stabilization. Thus, small-basis KANs provide a practical, corpus-transferable interface for auditing learned scalar transformations, but the tested replacements show no consistent benchmark, quality, or latency advantage over strong MLP baselines.

Abstract

Curse of Dimensionality

Explore similar work

Kolmogorov--Arnold Networks for Small Language Models

Multilevel Training for Kolmogorov Arnold Networks

Necessary and sufficient conditions for universality of Kolmogorov-Arnold networks