In-Context Symbolic Regression for Robustness-Improved Kolmogorov-Arnold Networks

Francesco Sovrano; Lidia Losavio; Giulia Vilone; Marc Langheinrich

cs.LGMar 16, 2026

In-Context Symbolic Regression for Robustness-Improved Kolmogorov-Arnold Networks

Authors: Francesco Sovrano, Lidia Losavio, Giulia Vilone, Marc Langheinrich

Organizations: University of Italian-Speaking Switzerland (USI) · Analog Devices International

Abstract

Symbolic regression aims to replace black-box predictors with concise analytical expressions that can be inspected and validated in scientific machine learning. Kolmogorov-Arnold Networks (KANs) are well suited to this goal because each connection between adjacent units (an "edge") is parametrised by a learnable univariate function that can, in principle, be replaced by a symbolic operator. In practice, however, symbolic extraction is a bottleneck: the standard KAN-to-symbol approach fits operators to each learned edge function in isolation, making the discrete choice sensitive to initialisation and non-convex parameter fitting, and ignoring how local substitutions interact through the full network. We study in-context symbolic regression for operator extraction in KANs, and present two complementary instantiations. Greedy in-context Symbolic Regression (GSR) performs greedy, in-context selection by choosing edge replacements according to end-to-end loss improvement after brief fine-tuning. Gated Matching Pursuit (GMP) amortises this in-context selection by training a differentiable gated operator layer that places an operator library behind sparse gates on each edge; after convergence, gates are discretised (optionally followed by a short in-context greedy refinement pass). We quantify robustness via one-factor-at-a-time (OFAT) hyper-parameter sweeps and assess both predictive error and qualitative consistency of recovered formulas. Across several experiments, greedy in-context symbolic regression achieves up to 99.8% reduction in median OFAT test MSE.

Figures & tables

Figure 1: Problem overview: isolated per-edge KAN-to-symbol fitting (AutoSym) is unstable and ignores end-to-end context.

Figure 2: Method overview: GSR selects operators by end-to-end loss improvement; GMP amortises in-context selection via sparse operator gates during training, then discretises (optionally refined by a short greedy pass) to reduce candidate-trial cost.

Feynman Dataset	AutoSym	FastKAN+AutoSym	GSR	FastKAN+GSR	GMP
I.13.4	1.31e3 $\pm$ 2.21e3	1.29e1 $\pm$ 1.77e1	7.35e-1 $\pm$ 3.94e-1	1.42e0 $\pm$ 6.61e-1	N/A
I.10.7	6.14e-2 $\pm$ 1.38e-2	8.49e-1 $\pm$ 8.64e-1	2.60e-3 $\pm$ 2.50e-3	1.90e-3 $\pm$ 3.00e-3	1.14e-2 $\pm$ 4.20e-3
I.12.1	1.07e2 $\pm$ 1.43e2	4.86e-1 $\pm$ 4.79e-1	7.18e-2 $\pm$ 2.25e-2	1.30e-2 $\pm$ 1.31e-2	1.25e0 $\pm$ 9.99e-1
II.34.29a	7.18e-4 $\pm$ 1.23e-4	8.60e-5 $\pm$ 1.07e-4	6.43e-4 $\pm$ 3.50e-4	5.99e-4 $\pm$ 3.77e-4	1.18e-2 $\pm$ 1.38e-2
I.9.18	1.40e-1 $\pm$ 2.15e-1	6.05e-3 $\pm$ 9.40e-3	2.89e-4 $\pm$ 1.53e-4	1.57e-3 $\pm$ 2.01e-3	N/A
I.12.4	4.64e-4 $\pm$ 3.70e-4	1.12e-4 $\pm$ 1.86e-4	1.24e-4 $\pm$ 2.50e-5	8.20e-5 $\pm$ 6.60e-5	1.73e-4 $\pm$ 3.60e-5

Table 1: Seed sensitivity at the reference configuration (width

[5,2]

\lambda=10^{-2}

, and three pruning cycles). We report test MSE as mean

\pm

std over available seeds for this fixed setting only. This table quantifies sensitivity to stochasticity at one chosen configuration; lower means and smaller standard deviations indicate greater robustness, but it is not an average of the OFAT sweep in Figure 3 . Entries marked with † use fewer than three successful seeds; for single-seed cases, only the observed value is reported. Entries marked N/A indicate that no successful run was obtained under the shared training/pruning schedule. Lower is better. The lowest mean is in bold ; the second-lowest mean is underlined .

Figure 3: OFAT hyper-parameter sensitivity distributions. Violins summarise test MSE across all valid one-factor-at-a-time runs obtained by varying hidden width,

\lambda

, and the number of pruning cycles around the reference configuration; dots denote individual observations. This figure aggregates hyper-parameter perturbations only and does not average over the seed-only repeats from Table 1 . Red

\times

markers indicate that a method produced no valid OFAT runs for that dataset and is therefore absent from the violin aggregation. Lower, tighter distributions indicate lower sensitivity and hence greater robustness.

Feynman Dataset	med (best)	med (AutoSym)	Reduction (%)
I.10.7	1.34e-3	7.36e-2	98.2
I.12.1	2.12e-2	9.49e0	99.8
I.13.4	6.43e-1	6.94e1	99.1
II.34.29a	3.05e-5	2.17e-4	85.9
II.21.32	3.25e-5	1.13e-3	97.1
II.6.15a	2.23e-3	2.58e-3	13.6

Table 2: Median OFAT test MSE of the best pipeline vs. AutoSym baseline, and percent reduction

100\cdot(1-\mathrm{med(best)}/\mathrm{med(AutoSym)})

Comparison ( $\text{best}\rightarrow\text{other}$ )	med (best)	med (other)	$p_{\mathrm{Holm}}$	Cliff’s $\delta$	CI
Feynman I.10.7 (best: FastKAN+GSR)
FastKAN+GSR $\rightarrow$ AutoSym (baseline)	1.34e-3	7.36e-2	1.89e-4 ***	-0.944	[4.51e-2, 1.55e0]
FastKAN+GSR $\rightarrow$ FastKAN+AutoSym	1.34e-3	4.52e-2	3.59e-3 **	-0.736	[1.35e-3, 1.84e0]
FastKAN+GSR $\rightarrow$ GMP	1.34e-3	1.06e-2	6.01e-3 **	-0.667	[-2.83e-4, 1.43e-2]
FastKAN+GSR $\rightarrow$ GSR	1.34e-3	2.38e-3	6.27e-2	-0.375	[-3.39e-3, 5.05e-3]
Feynman I.12.1 (best: FastKAN+GSR)

Table 3: One-sided Mann–Whitney

U

tests comparing the best pipeline (lowest median OFAT test MSE) to the others.

p

-values are Holm-corrected per dataset; effect size is Cliff’s

\delta

(negative favours the best pipeline). Bootstrap 95% CIs are reported for the median difference (other

-

best). Only methods with valid OFAT distributions are included; omitted rows correspond to unavailable distributions (notably selected GMP cases). Significance markers: * for

p_{\mathrm{Holm}}<0.05

, ** for

p_{\mathrm{Holm}}<0.01

, and *** for

p_{\mathrm{Holm}}<0.001

Explore similar work

Symbolic--KAN: Kolmogorov-Arnold Networks with Discrete Symbolic Structure for Interpretable Learning
Mar 25, 2026Salah A Faroughi, Farinaz Mostajeran, Amirhossein Arzani +1Interpretable ML Scientific ML
Energy & Intelligence Lab, Department of Chemical Engineering, University of Utah, Salt Lake City, Utah, 84112, USA · Department of Mechanical Engineering, University of Utah, Salt Lake City, Utah, 84112, USA · Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, Utah, 84112, USA +2
MorphingNet: Symbolic Regression Where the Network Is the Formula
Oct 14, 2026Jincheng Gao, Jiahe Yi, Shuo SunSymbolic Regression Feedforward Neural Networks
Thrust of Financial Technology, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.
Improving Neural Symbolic Regression with a Diverse Prior and Post-training Alignment
Oct 13, 2026David Otte, Jörg K. H. Franke, Frank HutterSymbolic Regression Synthetic Data Pretraining
LMU Munich · Munich Center for Machine Learning (MCML) · University of Freiburg +3

Mar 25, 2026J/K move · Enter open · S save

Symbolic--KAN: Kolmogorov-Arnold Networks with Discrete Symbolic Structure for Interpretable Learning

Salah A Faroughi, Farinaz Mostajeran, Amirhossein Arzani, Shirko Faroughi

Energy & Intelligence Lab, Department of Chemical Engineering, University of Utah, Salt Lake City, Utah, 84112, USA · Department of Mechanical Engineering, University of Utah, Salt Lake City, Utah, 84112, USA · Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, Utah, 84112, USA+2

Interpretable ML Scientific ML Kolmogorov-Arnold Networks Symbolic Regression PDE Discovery Physics-Informed ML

Open paper AskPDF arXiv

Symbolic discovery of governing equations is a long-standing goal in scientific machine learning, yet a fundamental trade-off persists between interpretability and scalable learning. Classical symbolic regression methods yield explicit analytic expressions but rely on combinatorial search, whereas neural networks scale efficiently with data and dimensionality but produce opaque representations. In this work, we introduce Symbolic Kolmogorov-Arnold Networks (Symbolic-KANs), a neural architecture that bridges this gap by embedding discrete symbolic structure directly within a trainable deep network. Symbolic-KANs represent multivariate functions as compositions of learned univariate primitives applied to learned scalar projections, guided by a library of analytic primitives, hierarchical gating, and symbolic regularization that progressively sharpens continuous mixtures into one-hot selections. After gated training and discretization, each active unit selects a single primitive and projection direction, yielding compact closed-form expressions without post-hoc symbolic fitting. Symbolic-KANs further act as scalable primitive discovery mechanisms, identifying the most relevant analytic components that can subsequently inform candidate libraries for sparse equation-learning methods. We demonstrate that Symbolic-KAN reliably recovers correct primitive terms and governing structures in data-driven regression and inverse dynamical systems. Moreover, the framework extends to forward and inverse physics-informed learning of partial differential equations, producing accurate solutions directly from governing constraints while constructing compact symbolic representations whose selected primitives reflect the true analytical structure of the underlying equations. These results position Symbolic-KAN as a step toward scalable, interpretable, and mechanistically grounded learning of governing laws.

Abstract

Explore similar work

Symbolic--KAN: Kolmogorov-Arnold Networks with Discrete Symbolic Structure for Interpretable Learning

MorphingNet: Symbolic Regression Where the Network Is the Formula

Improving Neural Symbolic Regression with a Diverse Prior and Post-training Alignment