Multilevel Training for Kolmogorov Arnold Networks

Ben S. Southworth; Jonas A. Actor; Graham Harper; Eric C. Cyr

cs.LGMar 5, 2026

Multilevel Training for Kolmogorov Arnold Networks

Authors: Ben S. Southworth, Jonas A. Actor, Graham Harper, Eric C. Cyr

Organizations: Theoretical Division, Los Alamos National Laboratory, USA. · Center for Computing Research, Sandia National Laboratories, USA.

Abstract

Algorithmic speedup of training common neural architectures is made difficult by the lack of structure guaranteed by the function compositions inherent to such networks. In contrast to multilayer perceptrons (MLPs), Kolmogorov-Arnold networks (KANs) provide more structure by expanding learned activations in a specified basis. This paper exploits this structure to develop practical algorithms and theoretical insights, yielding training speedup via multilevel training for KANs. To do so, we first establish an equivalence between KANs with spline basis functions and multichannel MLPs with power ReLU activations through a linear change of basis. We then analyze how this change of basis affects the geometry of gradient-based optimization with respect to spline knots. The KANs change-of-basis motivates a multilevel training approach, where we train a sequence of KANs naturally defined through a uniform refinement of spline knots with analytic geometric interpolation operators between models. The interpolation scheme enables a ``properly nested hierarchy'' of architectures, ensuring that interpolation to a fine model preserves the progress made on coarse models, while the compact support of spline basis functions ensures complementary optimization on subsequent levels. Numerical experiments demonstrate that our multilevel training approach can achieve orders of magnitude improvement in accuracy over conventional methods to train comparable KANs or MLPs, particularly for physics informed neural networks. Finally, this work demonstrates how principled design of neural networks can lead to exploitable structure, and in this case, multilevel algorithms that can dramatically improve training performance.

Figures & tables

Panel of Figure 1 — Figure 1: Eigenvalues of $(A^{[r]})^{T}A^{[r]}$ ordered smallest to largest (left) and the first (center) and last (right) five corresponding eigenvectors for $n=50$ splines knots and example orders $r\in\{1,3\}$ .

Panel of Figure 2 — Figure 2: Comparison of eigenvalue and eigenvector properties across spline orders.

Figure 3: Speedup of evaluating a layer, by applying the ReLU r activation and then the change-of-basis matrix, compared to computing the Cox-de Boor recursive formula for spline functions. Error bars show 1 standard deviation, computed over 10 instances.

Geometry	Gradient	Space	Prec. update
$\mathbf{w}$	$\nabla_{\mathbf{w}}\mathcal{L}$	$\mathbf{w}$	$\mathbf{w}_{k+1}=\mathbf{w}_{k}-\eta D_{\mathbf{w}}\nabla_{\mathbf{w}}\mathcal{L}$
$\mathbf{w}$	$\nabla_{\mathbf{w}}\mathcal{L}$	$\mathbf{u}$	$\mathbf{u}_{k+1}=\mathbf{u}_{k}-\eta A^{-T}D_{\mathbf{w}}A^{-1}\nabla_{\mathbf{u}}\mathcal{L}$
$\mathbf{w}$	$\nabla_{\mathbf{u}}\mathcal{L}$	$\mathbf{w}$	$\mathbf{w}_{k+1}=\mathbf{w}_{k}-\eta D_{\mathbf{u}}A\nabla_{\mathbf{w}}\mathcal{L}$
$\mathbf{w}$	$\nabla_{\mathbf{u}}\mathcal{L}$	$\mathbf{u}$	$\mathbf{u}_{k+1}=\mathbf{u}_{k}-\eta A^{-T}D_{\mathbf{u}}\nabla_{\mathbf{u}}\mathcal{L}$
$\mathbf{u}$	$\nabla_{\mathbf{u}}\mathcal{L}$	$\mathbf{w}$	$\mathbf{w}_{k+1}=\mathbf{w}_{k}-\eta A^{T}D_{\mathbf{u}}A\nabla_{\mathbf{w}}\mathcal{L}$
$\mathbf{u}$	$\nabla_{\mathbf{u}}\mathcal{L}$	$\mathbf{u}$	$\mathbf{u}_{k+1}=\mathbf{u}_{k}-\eta D_{\mathbf{u}}\nabla_{\mathbf{u}}\mathcal{L}$

Table 1: Preconditioned gradient descent update rules under different parameterizations and gradient representations. We associate the initial preconditioner to be consistent with the gradient, e.g.

\nabla_{\mathbf{u}}

has preconditioner

D_{\mathbf{u}}

Type	Layers	Basis	Fidelity	# Param.	MSE: mean (stdev)
KAN	$[2,5,1]$	ReLU	coarse	55	$1.10\times 10^{-2}\,(8.32\times 10^{-3})$
KAN	$[2,5,1]$	ReLU	fine	230	$1.35\times 10^{-0}\,(1.50\times 10^{-1})$
KAN	$[2,5,1]$	ReLU	multilevel	230	$1.06\times 10^{-2}\,(8.03\times 10^{-3})$
KAN	$[2,5,1]$	Spline	coarse	55	$1.65\times 10^{-3}\,(4.18\times 10^{-5})$
KAN	$[2,5,1]$	Spline	fine	230	$2.54\times 10^{-3}\,(5.64\times 10^{-3})$
KAN	$[2,5,1]$	Spline	multilevel	230	$\mathbf{3.67\times 10^{-5}}\,(7.19\times 10^{-5})$

Table 2: Accuracy for regression trained under different bases, architectures, and training regimes. Standard deviations are computed over random initializations (

N=5

Figure 4: Select convergence history for regression under approximately same amount of work for all different models. Vertical lines indicate refinements for multilevel models.

\begin{split}\mathcal{L}_{V}(\theta)&=\sum_{\vec{x}\in V}\bigl|\nabla\!\cdot\!(\epsilon(\vec{x})\,\nabla u_{N}(\vec{x}))\!-\!f(\vec{x})\bigr|^{2}\\ \mathcal{L}_{B}(\theta)&=\sum_{\vec{x}\in B}\bigl|u_{N}(\vec{x})\bigr|^{2}\\ \mathcal{L}_{I}(\theta)&=\sum_{\vec{x}\in I}\bigl|(\epsilon_{l}\!\nabla u_{N}(\vec{x}_{l})\!-\!\epsilon_{r}\!\nabla u_{N}(\vec{x}_{r}))\!\cdot\![1,0]\bigr|^{2}\\ \mathcal{L}(\theta)&=\gamma_{V}\,\mathcal{L}_{V}(\theta)+\gamma_{B}\,\mathcal{L}_{B}(\theta)+\gamma_{I}\,\mathcal{L}_{I}(\theta)\end{split}

Figure 5: For the 2d Possion problem in Sec. 5.2.1 (left) the volume, boundary and interface components of the PINN loss, and (right) multilevel KAN and MLP architectures and parameter counts. The neural network

u_{N}

has an assumed dependence on parameters

\theta

Figure 6: 2d Poisson: Plots of loss and error vs. epoch for multilevel training of KANs under a spline basis (left), multilevel training of KANs under a ReLU basis (center), and training of comparable MLP (right). The KAN Spline and KAN ReLU are identical in terms of approximation power, yet differences in multilevel training performance suggest that the relaxation for KAN Spline complements the refinement strategy.

Type	Layers	Fidelity	# Param.	Loss: mean (stdev)
KAN	$[2,20,20,1]$	coarse	3400	$6.635\times 10^{-3}\,(4.060\times 10^{-3})$
KAN	$[2,20,20,1]$	fine	9700	$4.072\times 10^{-3}\,(2.514\times 10^{-3})$
KAN	$[2,20,20,1]$	multilevel	9700	$\mathbf{2.402\times 10^{-5}}\,(1.897\times 10^{-5})$
MLP	$[2,20,20,1]$		500	$2.334\times 10^{-2}\,(1.740\times 10^{-3})$
MLP	$[2,56,56,1]$		3416	$1.734\times 10^{-2}(7.989\times 10^{-3})$

Table 3: Accuracy for PINN problem, comparing the effects of geometric refinement of the spline knots for the KAN basis. Comparable MLPs are included. Standard deviations computed across random initializations (

N=5

Figure 7: Training histories for multilevel refinement of equivalent KAN architectures in both ReLU and B-spline bases, with a similarly-sized MLP for comparison.

Figure 8: Plots of the trained B-spline-basis solution (leftmost) for the Allen-Cahn Equation in Section 5.2.3 , residual in log scale (left center), the Fourier transform of residual (right center), and cross-sections of the Fourier transform along the axes $\omega_{t}=0$ and $\omega_{x}=0$ (rightmost). Each row displays the output after training at each level of refinement, from coarsest (top) to finest (bottom). The first three columns share the same colorbar shown at the bottom of the column.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Plots of the trained ReLU-basis solution (leftmost) for the Allen-Cahn Equation, residual in log scale (left center), the Fourier transform of residual (right center), and cross-sections of the Fourier transform along the axes $\omega_{t}=0$ and $\omega_{x}=0$ (rightmost). Each row displays the output after training at each level of refinement, from coarsest (top) to finest (bottom). The first three columns share the same colorbar shown at the bottom of the column.

Explore similar work

Aug 1, 2026

SparseKAN: Compressing Kolmogorov--Arnold Networks Across Basis Functions, Neurons, and Bits

Kolmogorov--Arnold Networks (KANs) replace scalar edge weights with learnable univariate functions parameterized by multiple basis coefficients. This introduces a source of redundancy that conventional neural-network compression does not directly expose. We present SparseKAN, a unified approach that compresses KANs along three complementary axes: basis functions, neurons/channels, and numerical precision. SparseKAN equips the base branch, nonlinear basis branch, and individual basis terms with hierarchical learnable gates trained under a differentiable active-cost objective. The learned importance structure is subsequently hardened under explicit basis and width budgets, recovered in full or low precision, and physically compacted into smaller dense tensors rather than retained as sparse masks. Experiments on MNIST, CIFAR-10, and CIFAR-100 across spline, polynomial, RBF, wavelet, and convolutional KAN variants show that the structural axes compose predictably in cost. We also find strong basis-dependent differences in term importance: coefficient-based selection outperforms matched low-order truncation by up to 15.25 accuracy points in the evaluated Gram-polynomial settings. Eight-bit quantization is broadly robust, whereas 4-bit convolutional KANs require quantization-aware adaptation. Physical compaction removes up to 73.0% of parameters without accuracy loss on MNIST and reduces large-batch CUDA latency to as little as

0.51\times

dense execution. On a ZCU104 FPGA, the resulting sparse low-bit models achieve up to

23.63\times

lower inference latency, demonstrating that SparseKAN converts functional redundancy into measurable software and hardware efficiency. The SparseKAN implementation is available at https://github.com/OSU-STARLAB/SparseKAN.

Kazi Ahmed Asif Fuad, Lizhong Chen

Department of EECS Oregon State University Corvallis, OR 97331

Kolmogorov-Arnold Networks Neural Network Quantization Neural Network Pruning

Ask PDF

Oct 13, 2026

Hybrid Parallel Bessel KAN Model: a Kolmogorov Arnold Network: integrated framework for developing a novel architecture to reconstruct the fused-basis approach via a parameter free quadratic interaction

The Kolmogorov-Arnold Network (KAN) challenges traditional MLPs by placing learnable activation functions on network edges. However, scaling KANs to spatial data exposes limitations in default B-Splines. While hybrid architectures mix multiple basis families on a single edge to resolve this, monolithic superposition causes geometric overlap, parameter explosion, and severe numerical ill-conditioning (

κ(G) \approx 10^{16}

).To overcome this, we propose Parallel BesselKAN, which isolates B-Spline and low-order Bessel branches into parallel processing pipelines integrated via a parameter-free quadratic interaction. This decoupled architecture acts as an efficient internal ensemble, bypassing monolithic stability bottlenecks. It compresses the parameter footprint by

35\%

while achieving

97.99\%

accuracy on MNIST. Although standard MLPs retain a hardware speed advantage for pure vision tasks, Parallel BesselKAN provides a robust, parameter-efficient framework stabilizing mixed-basis networks and paving the way for scalable spatial adaptations like Convolutional KANs.

Mintu Pathak, Meshv Patel, Mohendra Roy +1

Department of Applied Sciences, Tezpur University, Napaam, Tezpur, 784028, Assam, India. · Department of ICT, Pandit Deendayal Energy University, Raysan, Gandhinagar, 382007, Gujarat, India.

Kolmogorov-Arnold Networks Modular Neural Networks

Ask PDF

Sep 1, 2026

RecKAN: Kolmogorov-Arnold Networks with a Learnable Recursive Polynomial Basis

Kolmogorov--Arnold Networks (KANs) replace the fixed scalar weights of a standard network with learnable univariate functions on each edge, but existing variants still fix the basis that those functions are built from: B-splines, Chebyshev polynomials, wavelets, or Jacobi polynomials, and learn only the combination weights over it. We introduce RecKAN, which instead defines the basis itself by a second order polynomial recurrence,

R_{n+1}(x) = (ax^2+bx+c)R_n(x) + (dx+e)R_{n-1}(x)

, whose five coefficients are learned jointly with the network. We show this recurrence recovers several classical polynomial families including both kinds of Chebyshev polynomials, Fibonacci, Pell, and Jacobsthal polynomials as special cases, and prove that its degree grows linearly in

n

exactly on the sub-family containing all of them, giving a concrete sense in which the learned basis can move beyond any fixed classical choice. Across multiple benchmark datasets spanning image, text, biomedical time series classification, and time series forecasting, RecKAN outperforms three parameter-matched KAN baselines (Chebyshev, Jacobi, and spline based) on all classification tasks and achieves the lowest MSE on the ETTh1 forecasting benchmark. Additionally, when used as a classifier head with a convolutional backbone, RecKAN achieves higher accuracy than standard MLP heads on Fashion MNIST, CIFAR-10, and SVHN. On a synthetic function fitting benchmark it tracks a sharply oscillatory target that a parameter comparable MLP under fits. We further show that the learned recurrence coefficients are interpretable: on the task requiring the most local structure, training moves the basis away from the linear degree growth regime that contains every classical family we identify, consistent with our theoretical analysis of what that structural shift enables.

Amirhosein Azarpour

Department of Computer Science Shahid Beheshti University Tehran, Iran

Polynomial Neural Networks Kolmogorov-Arnold Networks Neural Network Approximation Theory

Ask PDF