Kolmogorov-Arnold Networks: Curse of Dimensionality and Structural Bias

Yunhao Ni; Yuxin Guo; Jie Luo; Wenjun Wu; Lei Huang

cs.LGOct 15, 2026

Kolmogorov-Arnold Networks: Curse of Dimensionality and Structural Bias

Authors: Yunhao Ni, Yuxin Guo, Jie Luo, Wenjun Wu, Lei Huang

Organizations: State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

Abstract

Kolmogorov-Arnold networks (KANs) have a dimension-independent approximation exponent when the target admits a sufficiently smooth representation by univariate edge functions. We ask whether this favorable scaling belongs to the architecture or to the assumed target class. For arbitrary-depth standard spline-KANs, we prove a worst-case lower bound on isotropic Sobolev balls: below a dimension-independent accuracy threshold, the required number of spline coefficients grows at least exponentially with input dimension. Existing composition-matched MLP results, by contrast, show that favorable exponents on low-arity targets are not exclusive to KANs. We also revisit empirical comparisons. On four scaling-law targets, spline-KAN approximants constructed from known decompositions transfer exactly to MLPs with the cubic ReLU activation $σ(x)=(\max\{0,x\})^3$ , although their full dense parameter counts are larger. In controlled ridge-profile scans, KANs have a strong advantage when each nonlinear component depends on one input coordinate. When a component couples multiple input coordinates, this advantage may narrow sharply. At tight budgets, interacting targets often favor MLPs. In the random-label test on MNIST, ReLU-MLPs fit the randomized training labels more efficiently than spline-only KANs. Together, these results locate favorable scaling in target structure and a compatible inductive bias, not a general KAN-specific escape from the curse of dimensionality.

Figures & tables

Figure 1: Constructive scaling on four targets from Liu et al. (2025b) . Blue: constructed degree-three spline-KAN test RMSE versus spline-coefficient count. Purple: transferred ReLU 3 MLP test RMSE versus dense parameter count, at the validation-selected checkpoint (step zero or up to 200 Adam steps). At step zero the MLP represents the same function as its KAN source. All panels use

G\in\{8,16,32,64,128\}

Figure 2: Training-probe-selected test-MSE ratios on square-ridge targets with

d=16

and

R\in\{4,16,64\}

. Each family is selected independently within budget

P

. MLP candidates include four activations. The ratio

\rho>1

favors KAN, and

s

is the number of active coordinates per ridge. The full nine-panel scan is in Appendix C.3 .

Labels	Model	$P\leq 5\mathrm{k}$	$P\leq 20\mathrm{k}$	$P\leq 100\mathrm{k}$	$P\leq 500\mathrm{k}$
Standard	ReLU-MLP	87.5	94.9	98.0	98.6
Standard	Spline-only KAN	54.8	89.6	96.0	97.3
Random	ReLU-MLP	11.7	18.2	47.6	100.0
Random	Spline-only KAN	12.3	13.4	16.9	19.4

Table 1: MNIST accuracies (%) with

N_{\mathrm{train}}=60{,}000

within selected trainable-parameter budgets. Standard labels report test accuracy and random labels report final training accuracy. Each entry is the best architecture median over three seeds after selecting optimizer and learning rate by final training accuracy. Bold marks the higher median within each label condition and budget.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Constructive scaling on four targets from Liu et al. (2025b) . Blue: constructed cubic spline-KAN test RMSE versus spline-coefficient count. Purple: transferred ReLU 3 MLP test RMSE versus dense parameter count, at the validation-selected checkpoint (step zero or up to 200 Adam steps). At step zero the MLP represents the same function as its KAN source. All panels use

G\in\{8,16,32,64,128\}

Item	Setting
Target grid	$d,R\in\{4,16,64\}$ , and $s\in\{1,2,4,\ldots,d\}$
Target profiles	$\varphi(t)=t^{2}$ , $\sin(2\pi t)$ , $\tanh(t)$ , or $\operatorname{SiLU}(t)=t/(1+e^{-t})$
Hidden depth	$L\in\{2,4,6\}$ for both families
MLP	Width $w\in\{1,4,16,64,256\}$ ; ReLU, tanh, SiLU, or sine activation
Spline-only KAN	Width $w\in\{1,4,16,64\}$ ; $k=3$ ; $G\in\{3,7,15,31\}$ ; zero edge-wise base branch
Optimization	MSE; Adam; zero weight decay; 5,000 online-sampling steps; batch size 4,096; 500-step linear warmup followed by cosine decay to $1\%$ of the base rate

Appendix

Table 2: Controlled ridge-profile scan.

Figure 4: Complete square-ridge test-MSE ratios for

d,R\in\{4,16,64\}

, under the same training-probe selection rule as Figure 2 . The middle row is shown in the main text.

Figure 5: Sine-ridge test-MSE ratios under the same three-seed scan and parameter-budget selection as Figure 2 .

Figure 6: SiLU-ridge test-MSE ratios under the same scan and selection rule.

Figure 7: Tanh-ridge test-MSE ratios under the same scan and selection rule.

Figure 8: Sine-ridge test-error ratios when the MLP envelope is restricted to ReLU, tanh, or SiLU. Plotting and selection conventions follow Figure 5 for the

d\in\{4,16\}

subset. Removing sine activation expands the KAN-favored region, while denser supports still narrow that advantage.

Figure 9: Sine-ridge test-error ratios for a fixed ReLU-MLP comparator. Increasing support

s

again shifts the comparison toward MLP with a ReLU comparator.

Figure 10: Training-probe MSE ratios for the best-activation MLP comparison. The panels cover

d\in\{4,16\}

with the same selection rule as Figure 5 . The support-dependent pattern matches the independent-test ratios.

Figure 11: Absolute MSE of the models selected for the best-activation sine-ridge envelope. Solid curves report independent-test MSE and dashed curves report training-probe MSE. The dotted horizontal line marks MSE one, the zero-predictor baseline after target standardization.

Item	Setting
Hidden depth	$L\in\{2,4,6,8\}$
Hidden width	MLP: $w\in\{4,8,16,32,64,128,256,512\}$ ; KAN: $w\in\{1,2,4,8,16,32\}$
MLP	ReLU activation; Kaiming-uniform weights; zero biases
Spline-only KAN	$G=3$ , $k=3$ ; zero edge-wise base branch; spline coefficients trainable and spline scale fixed
Optimization	Cross-entropy, batch size 256, 200 epochs, no weight decay; SGD with momentum $0.9$ and learning rate in $\{0.1,0.03,0.01\}$ , or Adam with learning rate in $\{0.01,0.003,0.001\}$
Aggregation	For each architecture and seed, retain the optimizer/rate with highest final training accuracy; report medians and interquartile ranges over three seeds

Appendix

Table 3: MNIST classification and random-label fitting hyperparameters.

Figure 12: Full MNIST accuracy frontiers with

N_{\mathrm{train}}=60{,}000

. The left panel reports standard-label test accuracy and the right panel reports random-label final training accuracy. Each point is a three-seed architecture median after optimizer and learning-rate selection by final training accuracy. The curves show the best available value within each parameter budget.

Figure 13: Depth–width scan for normal-label testing (top) and random-label fitting (bottom). Cell values are median accuracies in percent. Width grids differ between MLP and KAN so that the scan covers overlapping parameter budgets. The plot title’s $N=60{,}000$ denotes the number of training examples. The random-label fitting gap persists across the tested depths.

Figure 14: Small-sample MNIST comparison with

N_{\mathrm{train}}=6{,}000

. The normal-label panel reports median test accuracy and the random-label panel reports median training accuracy over three seeds. The display is truncated just beyond the largest scanned KAN budget so that the visible range is covered by both families.

Curse of Dimensionality

See all 9 papers

Jul 17, 2026

Kolmogorov--Arnold Networks for Small Language Models

Kolmogorov--Arnold Networks (KANs) replace fixed node activations with learned one-dimensional edge functions, offering an explicit interface for interpretation and a possible alternative to transformer feed-forward networks. We test these claims separately. In a six-layer, 10M-parameter B-spline KAN, we reconstruct all 884,736 feed-forward edges: 87.8% exceed (NLS>0.1) and 0.4% are inactive. Pruning the lowest-activity 20--25% causes negligible loss increase, although structured MLP neuron pruning tolerates comparable sparsity. The audit replicates on BabyLM, but grid-size sweeps show that near-total fPCA compression and high closed-form-fit coverage are properties of the low-capacity grid-2 basis, not universal KAN behavior. For replacement, we evaluate MLP, SwiGLU, grouped Chebyshev, and rational GR-KAN networks on BabyLM. The KAN-family and gated variants improve validation loss over the GELU MLP, but this ordering does not transfer to standardized benchmarks: across ten seeds and 59,875 BLiMP pairs, accuracies span 62.4--63.1%, EWoK remains at chance, and a (+0.7)-point GR-KAN effect on BLiMP reverses on the supplement. Larger tests are also cautionary: parameter-matched MLPEdge underperforms the MLP on Wikitext-103, and 286M-parameter GR-KAN remains below a SwiGLU ClimbMix baseline after stabilization. Thus, small-basis KANs provide a practical, corpus-transferable interface for auditing learned scalar transformations, but the tested replacements show no consistent benchmark, quality, or latency advantage over strong MLP baselines.

Felippe Alves, Renato Vicente

TELUS Digital Research Hub – CIAAM Institute of Mathematics, Statistics and Computer Science University of São Paulo

Small Language Models Neural Network Interpretability Kolmogorov-Arnold Networks Feedforward Neural Networks

Ask PDF

Mar 5, 2026

Multilevel Training for Kolmogorov Arnold Networks

Algorithmic speedup of training common neural architectures is made difficult by the lack of structure guaranteed by the function compositions inherent to such networks. In contrast to multilayer perceptrons (MLPs), Kolmogorov-Arnold networks (KANs) provide more structure by expanding learned activations in a specified basis. This paper exploits this structure to develop practical algorithms and theoretical insights, yielding training speedup via multilevel training for KANs. To do so, we first establish an equivalence between KANs with spline basis functions and multichannel MLPs with power ReLU activations through a linear change of basis. We then analyze how this change of basis affects the geometry of gradient-based optimization with respect to spline knots. The KANs change-of-basis motivates a multilevel training approach, where we train a sequence of KANs naturally defined through a uniform refinement of spline knots with analytic geometric interpolation operators between models. The interpolation scheme enables a ``properly nested hierarchy'' of architectures, ensuring that interpolation to a fine model preserves the progress made on coarse models, while the compact support of spline basis functions ensures complementary optimization on subsequent levels. Numerical experiments demonstrate that our multilevel training approach can achieve orders of magnitude improvement in accuracy over conventional methods to train comparable KANs or MLPs, particularly for physics informed neural networks. Finally, this work demonstrates how principled design of neural networks can lead to exploitable structure, and in this case, multilevel algorithms that can dramatically improve training performance.

Ben S. Southworth, Jonas A. Actor, Graham Harper +1

Theoretical Division, Los Alamos National Laboratory, USA. · Center for Computing Research, Sandia National Laboratories, USA.

Kolmogorov-Arnold Networks Neural Network Optimization

Ask PDF

Apr 26, 2026

Necessary and sufficient conditions for universality of Kolmogorov-Arnold networks

We analyze the universal approximation property of Kolmogorov-Arnold Networks (KANs) in terms of their edge functions. If these functions are all affine, then universality clearly fails. How many non-affine functions are needed, in addition to affine ones, to ensure universality? We show that a single one suffices. More precisely, we prove that deep KANs in which all edge functions are either affine or equal to a fixed continuous function

σ

are dense in

C(K)

for every compact set

K\subset\mathbb{R}^n

if and only if

σ

is non-affine. In contrast, for KANs with exactly two hidden layers, universality holds if and only if

σ

is nonpolynomial. We further show that the full class of affine functions is not required; it can be replaced by a finite set without affecting universality. In particular, in the nonpolynomial case, a fixed family of five affine functions suffices when the depth is arbitrary. More generally, for every continuous non-affine function

σ

, there exists a finite affine family

A_σ

such that deep KANs with edge functions in

A_σ\cup\{σ\}

remain universal. We also prove that KANs with the spline-based edge parameterization introduced by Liu et al.~\cite{Liu2024} are universal approximators in the classical sense, even when the spline degree and knot sequence are fixed in advance.

Vugar Ismailov

Kolmogorov-Arnold Networks Universal Approximation Neural Network Approximation Theory Neural Network Expressivity

Ask PDF

Abstract

Curse of Dimensionality

Explore similar work

Kolmogorov--Arnold Networks for Small Language Models

Multilevel Training for Kolmogorov Arnold Networks

Necessary and sufficient conditions for universality of Kolmogorov-Arnold networks