Function Approximation

Follow this topic Ask about this topic

Latest papers

32 papers5 papers in the last four weeks, up 67% on the four weeks before. 0.0% of all new papers.

Oct 10, 2026

Description Complexity Bounds for Analytic Approximations of Lipschitz Functions

We study how many bits are needed to write down an elementary analytic expression that uniformly approximates a Lipschitz function. Expressions are finite trees over the basis

\{+,-,\times,÷,\exp\}

with the variables and rational constants, and their size is

\dc(f)

, the length of the shortest description of

f

in a fixed binary description language. For functions of one variable we prove that every

L

-Lipschitz

F:[0,1]\to[0,1]

has an everywhere-defined approximant of accuracy

Δ

with

\dc(f)\le C\,(L/Δ+\lg(1/Δ+2))

for all

L,Δ>0

, by a linear spline smoothed with logistic ramps in a description language with one repetition construct. We prove a matching lower bound valid for every description language: for

Δ\le 1/36

and every

L>0

some

L

-Lipschitz function needs at least

\tfrac1{16}(L/Δ+\lg(1/Δ+2))

bits. Hence the worst-case description complexity is

Θ(L/Δ+\lg(1/Δ+2))

. For functions on

[0,1]^d

the same method gives

Θ_d\bigl((L/Δ)^d+\lg(1/Δ+2)\bigr)

, using a tensor-product spline whose coefficients are mixed second differences and cost

O(d)

bits each. Without the repetition construct the upper bounds carry an additional logarithmic factor, whose necessity for plain expression trees is left open. A reference encoder, decoder and evaluator confirm the constructions numerically, and figures and tables report the checks.

Artem Vozmishchev

Faculty of Applied Informatics, ITMO University, St. Petersburg, 197101, Russia

Function Approximation

Oct 2, 2026

Bellman Error Minimization Via Linear Programming Normalization

This paper proposes a new functional approximation approach to reduce Bellman error in high-dimensional dynamic programming and Reinforcement Learning problems. Using a classic dynamic programming problem (network capacity control in revenue management) as the motivational example, the paper illustrates that deep neural networks and linear programming approximation algorithms can be combined to derive approximate solutions to dynamic programming problems. Simulation results show the proposed approximation algorithms achieves competitive performance when compared with benchmark.

Haining Yu

Independent Consultant

Function Approximation in RL Dynamic Programming Function Approximation Reinforcement Learning Markov Decision Processes

Sep 29, 2026

Kolmogorov-Arnold Classifier Systems as Universal Approximators

As the input dimension

n

grows, rule-based machine learning, such as Learning Classifier Systems (LCSs), faces a fundamental scalability bottleneck for function approximation: both rule count and parameter count grow exponentially with

n

. Traditional LCSs partition the

n

-dimensional input space directly, requiring

\mathcal{O}(m^n)

rules for adequate coverage, where

m

is the per-variable resolution. This article breaks from this paradigm by reorganizing rules dimension-wise, guided by the Kolmogorov-Arnold representation theorem: any continuous

n

-dimensional function can be expressed as a finite superposition of one-dimensional functions. The proposed Kolmogorov-Arnold Classifier System (KACS) decomposes the target function into one-dimensional subproblems and assigns a dedicated ruleset to each, reducing the worst-case rule count from

\mathcal{O}(m^n)

to

\mathcal{O}(mn^2)

and replacing

n

-dimensional local models with one-dimensional models requiring only two parameters per rule, independent of

n

. We also provide the first constructive proof that an LCS, namely KACS, is a universal approximator for continuous functions on compact domains. Evaluated against a direct

n

-dimensional input space partitioning approach under otherwise identical conditions, KACS achieves competitive accuracy in many settings while using only 2% to 40% of the parameters. Our implementation is available at https://github.com/YNU-NakataLab/KACS.

Hiroki Shiraishi, Hisao Ishibuchi, Masaya Nakata

Graduate School of Engineering Science, Yokohama National University, Yokohama 240-8501, Japan · Digital Healthcare Research Department, Hitachi, Ltd., Tokyo 185-8601, Japan · Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China +1

Universal Approximation Function Approximation Rule-Based Classification Curse of Dimensionality

Sep 15, 2026

A Weighted Kernel Method for Approximation that Adapts to Learned Multivariable Structure

Approximating the input-output behavior of a multivariable black-box function from limited data is challenging when blind to the importance of its inputs and their interactions. We introduce total sensitivity kernels (TSKs), a method based on families of weighted ANOVA kernels that learn and adapt to this multivariable structure. TSKs parameterize the weights on each multivariable component of the target function by factors for each input. We propose learning these factors directly from function evaluations by selecting the reproducing kernel Hilbert space (RKHS) in which the target function has minimum norm. Under suitable conditions, we show that this norm-minimization problem admits a unique solution, and we establish consistency of a finite-data formulation based on minimum-norm interpolation. The learned TSK factors characterize the participation of individual inputs across interactions and main effects, providing a kernel-dependent notion of input sensitivity related to total Sobol indices. Numerical experiments demonstrate that adapting the kernel to learned multivariable structure can substantially improve approximation accuracy over a standard product kernel.

John E. Darges, Laura Weidensager

Department of Mathematics, Emory University · Department of Mathematics, Simon Fraser University

Kernel Regression Kernel Methods Statistical Learning Theory Function Approximation

Sep 14, 2026

Stability-Constrained Approximation in Spline KANs: Exact Layer Balancing and Budget-Compatible Saturation

Deep spline superposition networks face a tension between approximation order and stability across depth. We study approximation under a hard layerwise Lipschitz budget, and organise it around two quantities: the factorisation stability complexity of a given deep factorisation, and the budget-compatible approximation complexity of a discretisation operator. First, we solve exactly the finite-depth diagonal balancing problem for a fixed chain of nonnegative envelope matrices: the optimal uniform layer budget equals

\|M_{L-1}\cdots M_0\|_{\infty\to\infty}^{1/L}

, attained by an explicit one-pass minimiser, for rectangular layers, with a complete treatment of degeneracies and non-attainment. The optimum can be arbitrarily larger than the Lipschitz constant of the network itself, because passing to envelopes destroys sign cancellation. Second, we give a constructive spline discretisation theorem preserving the budget up to a controlled slack, with an explicit grid threshold. Conversely, for linear spline-valued operators that preserve the budget exactly, we prove budget-compatible minimax lower bounds on classes constrained simultaneously in the first and third derivative norms -- a constraint pair that is forced by the problem and that rules out the usual scaling escapes. Finally, we show that the corresponding layer errors need not cancel under composition: for every operator of the class there is a stable depth-

L

tower realising a constant fraction of the accumulated error, so the linear-in-depth accumulation of the upper bound is not a proof artefact.

Aleksander Tankman

Fivestar Europe OÜ, Tallinn, Estonia

Function Approximation Neural Network Approximation Theory Kolmogorov-Arnold Networks Lipschitz Neural Networks

Sep 4, 2026

Shallow neural network approximation in mixed Sobolev spaces

We investigate the best

L_2

approximation of mixed Sobolev spaces by shallow neural networks with

n

neurons and general activation functions. We first establish an activation-independent Fourier-block principle: if an activation has univariate approximation order

ρ

in the sense of the Fourier-block property, then the global approximation rate has algebraic order

\min\{α,ρ\}

for target functions of mixed smoothness

α

, up to explicit logarithmic factors. To verify this property for concrete activations, we introduce a structured univariate approximation condition that implies the Fourier-block property with explicit parameters. For

\mathrm{ReLU}^k

, a matching algebraic lower bound identifies

\min\{α,k+1\}

as the optimal algebraic approximation exponent in any dimension, up to logarithmic factors in the upper bound. The framework also yields the exponent

\min\{α,k+1\}

for cardinal B-splines and soft-

\mathrm{ReLU}^k

, and the full mixed-smoothness exponent

α

for ELU and cosine activations, again up to logarithmic~factors.

Yuwen Li, Guozhi Zhang

School of Mathematical Sciences, Zhejiang University, 866 Yuhangtang Road, Hangzhou 310058, Zhejiang, China

Neural Network Approximation Theory Function Approximation Shallow Neural Networks Neural Network Activation Functions

Sep 3, 2026

SW-KAN: Kolmogorov-Arnold Networks with Stieltjes-Wigert q-Orthogonal Polynomials

Kolmogorov-Arnold Networks (KANs) represent a paradigmatic shift in deep learning by replacing fixed node activations with learnable univariate functions on edges, offering enhanced interpretability and parameter efficiency. While recent polynomial-based KAN variants have addressed the computational overhead of original B-spline implementations, they introduce a fundamental yet underexplored challenge: the domain mismatch between unbounded real-valued inputs and the bounded or semi-infinite support of orthogonal polynomial bases. To address this limitation, we propose the Stieltjes-Wigert Kolmogorov-Arnold Network (SW-KAN), a novel architecture that employs Stieltjes-Wigert q-orthogonal polynomials defined on the semi-infinite domain (0, infinity). We introduce a smooth exponential-of-tanh mapping that stably bridges the domain gap while preserving well-conditioned gradients, and leverage a numerically stable three-term recurrence that evaluates polynomial expansions in O(N) operations without special-function calls. Through comprehensive experiments spanning image classification and continuous function approximation, we demonstrate that SW-KAN achieves superior accuracy-efficiency trade-offs across diverse tasks. The log-normal weight structure and learnable q-parameter of Stieltjes-Wigert polynomials provide a distinct inductive bias that enables robust performance under resource-constrained conditions, including reduced feature dimensionality and limited training data. The proposed architecture not only outperforms established polynomial KAN baselines on standard benchmarks but also exhibits strong representational capacity for approximating complex multivariate functions with remarkably few parameters, making it a compelling alternative for efficient function approximation and classification in resource-constrained settings.

Amirhosein Azarpour, Seyyed Moein Kazemi

Faculty of Mathematical Sciences, Department of Computer Science, Shahid Beheshti University, Tehran · Faculty of Mathematical Sciences, Shahid Beheshti University, Tehran

Function Approximation Kolmogorov-Arnold Networks Polynomial Neural Networks Neural Network Approximation Theory

Aug 21, 2026

Resolution-Consistent Greedy Neural Approximation on Infinite-Dimensional Spaces

We develop constructive approximation and learning guarantees for shallow neural models with infinite-dimensional inputs observed through finitely many coordinates. The analysis is based on a parameter-normalized neural dictionary and its associated weighted variation class. Within this class, the approximation error separates into a distribution-dependent coordinate-truncation term and a greedy finite-width term. For empirical regression, a fully-corrective greedy procedure yields population guarantees whose statistical complexity is uniform in the retained input resolution. The same framework extends to Hilbert-valued responses without an explicit dependence on the output dimension. The dimension-free statements are statistical, not computational: selecting a new neuron still requires solving a nonconvex parameter-search problem. The quasi-Polish construction underlying recent infinite-dimensional universal approximation results provides a motivating example, and synthetic experiments illustrate the predicted resolution, width, and sample-size regimes.

Pablo M. Berná, Antonio Falcó, Diego Mondéjar

Universidad Tecnol´ogica Atl´antico Mediterr´aneo – UTAMED, Facultad de Empresa Digital, Tecnolog´ıa y Derecho, M´alaga, Espa˜na · Departamento de Matem´aticas, F´ısica y Ciencias Tecnol´ogicas, Universidad Cardenal Herrera-CEU, CEU Universities, San Bartolom´e 55, Alfara del Patriarca (Valencia), 46115, Spain · Departamento de Matem´aticas, CUNEF Universidad, 28040, Espa˜na

Neural Network Approximation Theory Function Approximation Shallow Neural Networks Statistical Learning Theory

Aug 11, 2026

Decision-Aware Approximation of Belief Functions for Evidential Combinatorial Optimization

Reducing the number of focal elements of a mass function is classically driven by an intrinsic distance, such as Jaccard or Jousselme, that keeps the approximation close to the original as a body of evidence. We consider instead the case where the mass function feeds a linear combinatorial optimisation problem with evidential costs. What should then be preserved is not the closeness of the two mass functions, but the quality of the decision they induce. We introduce a decision-aware approximation that targets the regret of the decision: one decides with the cheaper approximation and is evaluated under the true mass function. On a minimal shortest path, the distance-optimal approximation flips the decision while a decision-aware merge preserves it, and this occurs on a non-negligible fraction of random instances. We prove a one-point bound that localises the regret at the true optimum, turn it into an exact dynamic program for the scalar case, and extend it to an online version that prunes focal elements before the final cost is known. In experiments the decision-aware compressor flips the decision less often than representation-aware compression, for both the linear criterion and a non-linear proxy read-out.

Sohaib Afifi

Univ. Artois, UR 3926, Laboratoire de Génie Informatique et d’Automatique de l’Artois (LGI2A), Béthune, France

Dempster-Shafer Theory Combinatorial Optimization Decision-Making under Uncertainty Function Approximation

Aug 2, 2026

Do Neural Networks Really Beat the Curse of Dimensionality? A Bit-Complexity View

Traditional approximation theory measures convergence rates in terms of the number of parameters or degrees of freedom. However, practical computation operates under finite precision: parameters must be encoded using a finite number of bits. Therefore, approximation efficiency should be evaluated in terms of computational bit complexity, which is intrinsically connected to the metric entropy of the underlying function class. In this work, we develop a unified approximation framework based on binary encoding and metric entropy. We analyze classical methods (including polynomial approximation, sparse grids, and finite elements) as well as shallow and deep neural networks, and compare their approximation rates for function classes with comparable metric entropy. We observe that, when evaluated in terms of bits, most classical methods are in general suboptimal relative to the intrinsic limits dictated by metric entropy, while neural network methods may exhibit different behaviors. We show that when complexity is measured in bits rather than parameters, no method fundamentally exceeds the approximation order achieved by classical approaches. Our results also indicate that many seeming advantages of neural networks, including dimension-independent rates and superconvergence phenomena, stem from differences in function class complexity rather than intrinsic architectural superiority. In this sense, the traditional curse of dimensionality can be misleading; the fundamental limitation is instead a curse of bit complexity, governed by metric entropy.

Tong Mao, Jinchao Xu

King Abdullah University of Science and Technology, Thuwal 23955, Saudi Arabia

Curse of Dimensionality Neural Network Approximation Theory Function Approximation

Jul 10, 2026

All you need is SAMPAT

The current state of the art in AI/ML rests on deep neural architectures, which, in general, suffer from a lack of interpretability. Interpretability is crucial to gleaning insights while analyzing experimental data, where quantitative predictions may not be adequate for a scientist. We present a three layer neural architecture, SAMPAT (Smooth Approximation via Multivariate Polynomials and Analytic Transformations), that can provably learn a continuous, everywhere differentiable function, that can approximate any smooth function arbitrarily closely. SAMPAT's approximant can be expressed as a closed and compact algebraic, analytic expression, providing complete interpretability. Experiments on synthetic and benchmark datasets indicate that SAMPAT yields competitive performance with simpler representations. For many tasks, a two layer SAMPAT suffices. By imposing restrictions on the connectivity between neurons, SAMPAT may be used to provide a range of approximants, including regular and trigonometric polynomials, rational expressions, Gaussians, mixtures of Gaussians, as well as arbitrary combinations of the same; without restrictions, it learns a suitable structure. SAMPAT may be used to factorize polynomials and model nonlinear systems. With the addition of skip connections, a 4 to 6 layer SAMPAT is adequate to represent a substantive range of methods widely used in AI/ML, allowing the choice of a model's family, not just its parameters, to also be optimized as part of the learning process.

Jayadeva, Madhur Aswani

Department of Electrical Engineering, Indian Institute of Technology, Delhi

Shallow Neural Networks Interpretable ML Neural Network Interpretability Neural Network Approximation Theory Neural Network Expressivity Function Approximation

Jun 25, 2026

Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation

Feedforward neural network (NN) expressivity is typically studied by emulating optimal basis-expansion schemes. While powerful, this perspective is incomplete: it primarily captures complexity through regularity, and therefore does not distinguish intuitively simple and complicated objects with comparable regularity, such as the square-root function and a typical Brownian path. The guiding message is that neural networks should be viewed not only as flexible basis functions, but also as models of computation. If a function is computable by a real-valued circuit over a prescribed elementary gate language, then it can be computed to comparable accuracy by an NN with explicit depth, width, and non-zero-parameter bounds controlled by the depth, width, gate count, and gate structure. Thus, neural-network complexity is not governed by regularity alone, but also by algorithmic complexity. We then show that any definable NN model satisfying a natural parallelization condition, allowing possibly multivariate non-linearities such as attention or layer normalization, is a universal approximator if and only if it contains a non-affine nonlinearity. The scope of our theory is illustrated by deducing universal approximation guarantees for continuous functions, minimax-optimal approximation guarantees for Besov classes, logarithmic-error complexity for holomorphic functions, and by showing that NNs can emulate numerical algorithms such as Newton-Raphson root finding and power iteration without architecture-specific arguments. Its precision is illustrated by shortest-path computation on

k

-vertex graphs: compiling the tropical dynamic-programming circuit yields NNs with O(log(1/ε)) non-zero parameters, exponentially improving in 1/ε over the generic

O(ε^{-c k^2})

Lipschitz-approximation scale, for a constant c>0.

Anastasis Kratsios, Simone Brugiapaglia, Bum Jun Kim +2

McMaster University & Vector Institute, Toronto, ON, Canada · Concordia University, Montréal, QC, Canada · University of Tokyo, Tokyo, Japan +2

Universal Approximation Neural Network Approximation Theory Feedforward Neural Networks Neural Network Expressivity Function Approximation

Jun 22, 2026

EML Trees Are Universal Approximators

The recently introduced EML (Exp-Minus-Log) function acts as continuous analogue of NAND gates, providing a compositional building block capable of representing elementary functions. In this work, we study the expressive power of tree-structured compositions of EML functions. We show that such trees enjoy a universal approximation property for functions in

W^{k, \infty}

for

k \in \mathbb N

, drawing on classical neural network approximation arguments while exploiting the ability to explicitly construct EML trees that mimic polynomial representations. We further propose a learning algorithm for EML-type trees equipped with fitting parameters, and demonstrate its feasibility in practical optimization problems. Our results establish EML trees as a theoretically grounded framework for function approximation.

Joe Germany, Elie Abdo, Joseph Bakarji

Department of Mathematics American University of Beirut Beirut, Lebanon · Department of Mechanical Engineering American University of Beirut Beirut, Lebanon

Universal Approximation Neural Network Approximation Theory Neural Network Expressivity Function Approximation

Jun 18, 2026

Recurrent neural networks approximate continuous functions

Classical approximation theorems ask for a new neural network whenever the target accuracy is improved. This paper studies the opposite possibility: can the network be chosen once and for all, and can accuracy be bought only by letting it run longer? We prove that this is possible for every continuous function on [-1,1]. More precisely, each such function is uniformly approximated by the time evolution of a single ReLU recurrent neural network with fixed weights and fixed hidden dimension. The mechanism behind the construction is a new intermediate model, the Turing machine with neural units (TMNU). This model retains the algorithmic freedom needed to implement polynomial approximation schemes, while remaining rigid enough to be simulated by RNNs with explicit bounds on hidden dimension and weight magnitude. The resulting convergence rates reflect the underlying polynomial approximation rates. We complement the construction with minimax lower bounds showing that runtime is not merely a proof artifact, but an unavoidable resource in this fixed-network approximation paradigm.

Valentin Abadie, Clemens Hutter, Helmut Bölcskei

ReLU Neural Networks Recurrent Neural Networks Neural Network Approximation Theory Function Approximation

Jun 16, 2026

A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noise

Temporal difference (TD) learning with linear function approximation is a core method for policy evaluation. Its classical continuous-time description is an ordinary differential equation (ODE), which captures the asymptotic mean dynamics but neglects stochastic fluctuations determining the error floor. We introduce a stochastic differential equation (SDE) approximation for linear TD(0) under Markovian noise. The resulting model distinguishes the contraction dynamics governed by the projected Bellman operator from the influence of Markovian sampling. As a consequence, the model explains the constant-stepsize error floor through the interaction between Markovian long-run covariance and the contraction geometry of the projected Bellman operator.

M. Forzo, E. Monzio Compagnoni, A. Russo +1

Technical University of Munich (TUM), Munich, Germany · University of Basel, Basel, Switzerland · Boston University, Boston, USA.

Stochastic Differential Equations Temporal-Difference Learning Linear Function Approximation Function Approximation in RL Function Approximation

Jun 7, 2026

Compositional Approximation Can Strictly Outperform Superpositional Approximation

Many classically studied function classes are known to be approximated optimally by superpositional methods, i.e. with approximants constructed as the linear combination of elements in some dictionary. Here optimality means that the uniform approximation error viewed as a function of the number of parameters used has polynomial decay of the highest order achievable by any parametrized method whose parameters can be encoded as a bit string of length proportional, up to logarithmic factors, to the number of parameters. While compositional methods like neural networks are structurally different, their approximation rates can be made comparable by imposing constraints that ensure such a proportional bit string encoding. In this work we study function classes exhibiting structural properties that limit superpositional approximation rates to be strictly lower than compositional approximation rates. In particular, we construct explicit examples for which there is an arbitrarily large gap.

Dennis Elbrächter, Philipp Petersen

Faculty of Mathematics, University of Vienna, Austria

Neural Network Approximation Theory Function Approximation

Jun 3, 2026

Fitting scattered data with optional monotonicity constraints on GPU: LipFit package

This paper presents a method of multivariate scattered data interpolation and approximation that produces optimal Lipschitz-continuous approximation, subject to the desired monotonicity constraints. This method relies on tight upper and lower approximations to the data, and is similar in its spirit to the nearest-neighbour approximation but does not suffer from discontinuities. Local Lipschitz interpolation and Lipschitz smoothing are also presented. This approach falls under the umbrella of instance-based approximation with no training phase, and it is suitable for GPU-based parallelisation. A Python GPU-friendly package LipFit which implements the methods discussed is discussed.

Gleb Beliakov

School of Information Technology, Deakin University, Burwood 3125, Australia

Function Approximation

May 20, 2026

Approximation Theory for Neural Networks: Old and New

Universal approximation theorems provide a mathematical explanation for the expressive power of neural networks. They assert that, under mild conditions on the activation function, feedforward neural networks are dense in broad function classes, such as continuous functions on compact subsets of

\mathbb{R}^d

,

L^p

spaces, or Sobolev spaces. Over the past four decades, these qualitative universality results have evolved into a rich quantitative theory addressing approximation rates, parameter efficiency, and the role of architectural features such as depth and width. This survey presents several glimpses into this theory. We review classical density results for single-hidden-layer networks, as well as quantitative bounds that relate approximation error to network size and smoothness assumptions on target functions. Particular emphasis is placed on depth--width trade-offs and on results demonstrating that deeper architectures can achieve superior parameter efficiency for structured function classes. In addition to standard feedforward neural networks, we also review recent developments on Kolmogorov--Arnold Networks (KANs), which offer an alternative architectural paradigm and whose approximation-theoretic properties have begun to attract significant theoretical attention.

Soumendu Sundar Mukherjee, Himasish Talukdar

Statistics and Mathematics Unit, Indian Statistical Institute, Kolkata

Universal Approximation Neural Network Approximation Theory Feedforward Neural Networks Neural Network Expressivity Function Approximation

May 20, 2026

Adaptive RBF-KAN: A Comparative Evaluation of Dynamic Shape Parameters in Kolmogorov-Arnold Networks

Kolmogorov-Arnold Networks (KANs) approximate multivariate functions using learnable univariate edge functions, typically parameterized by B-spline bases. Although effective, spline-based implementations can be computationally expensive. A modified version of KANs, called FastKAN, improves efficiency by replacing splines with Gaussian radial basis functions (RBFs), but it relies on a fixed kernel and shape parameter. In this work, we extend the RBF-based KAN framework by introducing a broader family of radial basis kernels and by initializing the kernel shape parameter using leave-one-out cross-validation (LOOCV). To the best of our knowledge, this is the first study that integrates LOOCV-based kernel scale estimation with deep KAN training. We also introduce Matérn and Wendland kernels into the KAN framework for the first time, enabling more flexible basis representations beyond the Gaussian kernel used in FastKAN. The LOOCV estimate provides a data-driven initialization of the kernel scale, which is subsequently refined during network training. The proposed adaptive RBF-KAN is evaluated on several two-dimensional benchmark functions. The results highlight the importance of kernel selection and adaptive shape parameters, with different kernels showing advantages for smooth functions, discontinuities, and oscillatory patterns. Overall, combining LOOCV-based initialization with adaptive kernel learning provides a practical strategy for improving RBF-based KAN models.

Roberto Cavoretto, Alessandra De Rossi, Adeeba Haider +1

Department of Mathematics “Giuseppe Peano”, University of Torino, via Carlo Alberto 10, 10123 Torino, Italy · Member of the INdAM Research Group GNCS · Department of Mathematics, Hong Kong Baptist University, Hong Kong SAR, China

Kolmogorov-Arnold Networks Kernel Methods Function Approximation Radial Basis Function Networks

May 15, 2026

Thermodynamic Networks: Harnessing Non-Equilibrium Steady States for Computation

We introduce thermodynamic networks, a general framework for autonomous, physics-based computation using non-equilibrium steady states. These networks are modeled as a collection of finite-size reservoirs that exchange conserved quantities--such as electric charge or molecular number--while relaxing to a non-equilibrium steady state, which encodes the solution of a computational problem. We identify Negative Differential Conductance (NDC) as the critical physical property governing the computational expressivity of the thermodynamic network. While networks lacking NDC are restricted to computing monotonic functions, the presence of NDC enables universal function approximation. For the training of the network, we use protocols that take advantage of the natural tendency of the system to equilibrate. We illustrate the versatility of our approach via two different platforms: quantum dot networks and enzymatic reaction networks. Both systems can be engineered to have NDC, enabling high performance in standard benchmarks, including sine function approximation and MNIST digit classification. Overall, our work establishes a rigorous link between non-equilibrium steady states and computational expressivity.

Patryk Lipka-Bartosik, Gianmichele Blasi, Javier Lalueza Puértolas +3

Center for Theoretical Physics, Polish Academy of Sciences, Warsaw, Poland · Instituto de Física Interdisciplinar y Sistemas Complejos IFISC (CSIC-UIB), E-07122 Palma de Mallorca, Spain · Física Teòrica: Informació i Fenòmens Quàntics, Department de Física, Universitat Autònoma de Barcelona, 08193 Bellaterra (Barcelona), Spain +1

Thermodynamic Computing Function Approximation

Apr 27, 2026

Transformer Approximations from ReLUs

We provide a systematic recipe for translating ReLU approximation results to softmax attention mechanism. This recipe covers many common approximation targets. Importantly, it yields target-specific, economic resource bounds beyond universal approximation statements. We showcase the recipe on multiplication, reciprocal computation, and min/max primitives. These results provide new analytical tools for analyzing softmax transformer models.

Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee +1

Center for Foundation Models and Generative AI, Northwestern University, Evanston, IL 60208, USA · Department of Computer Science, Northwestern University, Evanston, IL 60208, USA · Department of Physics, National Taiwan University, Taipei 10617, Taiwan +1

Softmax Attention Neural Network Approximation Theory Attention Mechanisms Attention Approximation Function Approximation

Mar 30, 2026

Universal Approximation Constraints of Narrow ResNets: The Tunnel Effect

We analyze the universal approximation constraints of narrow Residual Neural Networks (ResNets) both theoretically and numerically. For deep neural networks without input space augmentation, a central constraint is the inability to represent critical points of the input-output map. We prove that this has global consequences for target function approximations and show that the manifestation of this defect is typically a shift of the critical point to infinity, which we call the ``tunnel effect'' in the context of classification tasks. While ResNets offer greater expressivity than standard multilayer perceptrons (MLPs), their capability strongly depends on the signal ratio between the skip and residual channels. We establish quantitative approximation bounds for both the residual-dominant (close to MLP) and skip-dominant (close to neural ODE) regimes. These estimates depend explicitly on the channel ratio and uniform network weight bounds. Low-dimensional examples further provide a detailed analysis of the different ResNet regimes and how architecture-target incompatibility influences the approximation error.

Christian Kuehn, Sara-Viola Kuntz, Tobias Wöhrer

Technical University of Munich, School of Computation, Information and Technology, Department of Mathematics, Boltzmannstraße 3, 85748 Garching, Germany · Munich Data Science Institute (MDSI), Garching, Germany · Munich Center for Machine Learning (MCML), München, Germany +1

Universal Approximation Neural Network Approximation Theory Function Approximation Neural Network Expressivity Residual Learning

Mar 21, 2026

CFNN: Continued Fraction Neural Network

Accurately characterizing non-linear functional manifolds with singularities is a fundamental challenge in scientific computing. While Multi-Layer Perceptrons (MLPs) dominate, their spectral bias hinders resolving high-curvature features without excessive parameters. We introduce Continued Fraction Neural Networks (CFNNs), integrating continued fractions with gradient-based optimization to provide a

rational inductive bias.'' This enables capturing complex asymptotics and discontinuities with extreme parameter frugality. We provide formal approximation bounds demonstrating exponential convergence and stability guarantees. To address recursive instability, we develop three implementations: CFNN-Boost, CFNN-MoE, and CFNN-Hybrid. Benchmarks show CFNNs consistently outperform MLPs in precision with one to two orders of magnitude fewer parameters, exhibiting up to a 47-fold improvement in noise robustness and physical consistency. By bridging black-box flexibility and white-box transparency, CFNNs establish a reliable

grey-box'' paradigm for AI-driven scientific research.

Chao Wang, Xuancheng Zhou, Ruilin Hou +2

School of Future Technology, Shanghai University, Shanghai, 200444, China. · Institute of Artificial Intelligence, Shanghai University, Shanghai, 200444, China. · School of Computer Science and Technology, Shanghai University, Shanghai, 200444, China. +2

Function Approximation Scientific ML Neural Network Approximation Theory

Mar 17, 2026

Online Semi-infinite Linear Programming: Efficient Algorithms via Function Approximation

We consider the dynamic resource allocation problem where the decision space is finite-dimensional, yet the solution must satisfy a large or even infinite number of constraints revealed via streaming data or oracle feedback. We model this challenge as an Online Semi-infinite Linear Programming (OSILP) problem and develop a novel LP formulation to solve it approximately. Specifically, we employ function approximation to reduce the number of constraints to a constant

q

. This addresses a key limitation of traditional online LP algorithms, whose regret bounds typically depend on the number of constraints, leading to poor performance in this setting. We propose a dual-based algorithm to solve our new formulation, which offers broad applicability through the selection of appropriate potential functions. We analyze this algorithm under two classical input models-stochastic input and random permutation-establishing regret bounds of

O(q\sqrt{T})

and

O\left(\left(q+q\log{T})\sqrt{T}\right)\right)

respectively. Note that both regret bounds are independent of the number of constraints, which demonstrates the potential of our approach to handle a large or infinite number of constraints. Furthermore, we investigate the potential to improve upon the

O(q\sqrt{T})

regret and propose a two-stage algorithm, achieving

O(q\log{T} + q/ε)

regret under more stringent assumptions. We also extend our algorithms to the general function setting. A series of experiments validates that our algorithms outperform existing methods when confronted with a large number of constraints.

Yiming Zong, Jiashuo Jiang

Department of Industrial Engineering & Decision Analytics, Hong Kong University of Science and Technology

Function Approximation Linear Programming Online Resource Allocation

Mar 16, 2026

Deep learning and the rate of approximation by flows

We investigate the dependence of the approximation capacity of deep residual networks on its depth in a continuous dynamical systems setting. This can be formulated as the general problem of quantifying the minimal time-horizon required to approximate a diffeomorphism by flows driven by a given family

\mathcal F

of vector fields. We show that this minimal time can be identified as a geodesic distance on a sub-Finsler manifold of diffeomorphisms, where the local geometry is characterised by a variational principle involving

\mathcal F

. This connects the learning efficiency of target relationships to their compatibility with the learning architectural choice. Further, the results suggest that the key approximation mechanism in deep learning, namely the approximation of functions by composition or dynamics, differs in a fundamental way from linear approximation theory, where linear spaces and norm-based rate estimates are replaced by manifolds and geodesic distances.

Jingpu Cheng, Qianxiao Li, Ting Lin +1

Department of Mathematics, National University of Singapore, 117543, Singapore · Department of Mathematics and Institute for Functional Intelligent Materials, National University of Singapore, 117543, Singapore · School of Mathematical Sciences, Peking University, 100871, China

Neural Network Approximation Theory Function Approximation Residual Learning

Mar 10, 2026

Global universality via discrete-time signatures

We establish global universal approximation theorems for non-anticipative and general path-dependent functionals on spaces of piecewise linear paths, stating that linear functionals of the corresponding signatures are dense with respect to

L^p

- and weighted norms. We verify that these approximation results are applicable to piecewise linear interpolations of a class of Gaussian processes, including Brownian motion and fractional Brownian motion with Hurst parameter

H>1/4

. Moreover, we derive quantitative convergence rates for signatures of piecewise linear approximations of Gaussian processes towards their continuous-time counterparts. Consequently, we obtain

L^p

-approximation results for path-dependent functionals of Gaussian processes, as well as for random ordinary differential equations and stochastic differential equations driven by Brownian motion.

Mihriban Ceylan, David J. Prömel

Path Signatures Universal Approximation Function Approximation

Mar 9, 2026

DualFlexKAN: Dual-stage Kolmogorov-Arnold Networks with Independent Function Control

Multi-Layer Perceptrons (MLPs) rely on pre-defined, fixed activation functions, imposing a static inductive bias that forces the network to approximate complex topologies solely through increased depth and width. Kolmogorov-Arnold Networks (KANs) address this limitation through edge-centric learnable functions, yet their formulation suffers from quadratic parameter scaling and architectural rigidity that hinders the effective integration of standard regularization techniques. This paper introduces the DualFlexKAN (DFKAN), a flexible architecture featuring a dual-stage mechanism that independently controls pre-linear input transformations and post-linear output activations. This decoupling enables hybrid networks that optimize the trade-off between expressiveness and computational cost. Unlike standard formulations, DFKAN supports diverse basis function families, including orthogonal polynomials, B-splines, and radial basis functions, integrated with configurable regularization strategies that stabilize training dynamics. Comprehensive evaluations across regression benchmarks, physics-informed tasks, and function approximation demonstrate that DFKAN outperforms both MLPs and conventional KANs in accuracy, convergence speed, and gradient fidelity. The proposed hybrid configurations achieve superior performance with one to two orders of magnitude fewer parameters than standard KANs, effectively mitigating the parameter explosion problem while preserving KAN-style expressiveness. DFKAN provides a principled, scalable framework for incorporating adaptive non-linearities, proving particularly advantageous for data-efficient learning and interpretable function discovery in scientific applications.

Andrés Ortiz, Nicolás J. Gallego-Molina, Carmen Jiménez-Mesa +2

Department of Communications Engineering University of Malaga Malaga, Spain · DaSCI Andalusian Institute of Data Science and Computational Intelligence University of Granada Granada, Spain · Department of signal Theory, Communications and Telematics University of Granada Granada, Spain

Function Approximation Kolmogorov-Arnold Networks Neural Network Approximation Theory

Feb 20, 2026

Learning Without Training

Machine learning is at the heart of managing the real-world problems associated with massive data. With the success of neural networks on such large-scale problems, more research in machine learning is being conducted now than ever before. This dissertation focuses on three different projects rooted in mathematical theory for machine learning applications. The first project deals with supervised learning and manifold learning. In theory, one of the main problems in supervised learning is that of function approximation: that is, given some data set

\mathcal{D}=\{(x_j,f(x_j))\}_{j=1}^M

, can one build a model

F\approx f

? We introduce a method which aims to remedy several of the theoretical shortcomings of the current paradigm for supervised learning. The second project deals with transfer learning, which is the study of how an approximation process or model learned on one domain can be leveraged to improve the approximation on another domain. We study such liftings of functions when the data is assumed to be known only on a part of the whole domain. We are interested in determining subsets of the target data space on which the lifting can be defined, and how the local smoothness of the function and its lifting are related. The third project is concerned with the classification task in machine learning, particularly in the active learning paradigm. Classification has often been treated as an approximation problem as well, but we propose an alternative approach leveraging techniques originally introduced for signal separation problems. We introduce theory to unify signal separation with classification and a new algorithm which yields competitive accuracy to other recent active learning algorithms while providing results much faster.

Ryan O'Dowd

Claremont Graduate University

Function Approximation Supervised Learning Transfer Learning Active Learning Manifold Learning

Feb 18, 2026

FEKAN: Feature-Enriched Kolmogorov-Arnold Networks

Kolmogorov-Arnold Networks (KANs) have recently emerged as a compelling alternative to multilayer perceptrons, offering enhanced interpretability via functional decomposition. However, existing KAN architectures, including spline-, wavelet-, radial-basis variants, etc., suffer from high computational cost and slow convergence, limiting scalability and practical applicability. Here, we introduce Feature-Enriched Kolmogorov-Arnold Networks (FEKAN), a simple yet effective extension that preserves all the advantages of KAN while improving computational efficiency and predictive accuracy through feature enrichment, without increasing the number of trainable parameters. By incorporating these additional features, FEKAN accelerates convergence, increases representation capacity, and substantially mitigates the computational overhead characteristic of state-of-the-art KAN architectures. We investigate FEKAN across a comprehensive set of benchmarks, including function-approximation tasks, physics-informed formulations for diverse partial differential equations (PDEs), and neural operator settings that map between input and output function spaces. For function approximation, we systematically compare FEKAN against a broad family of KAN variants, FastKAN, WavKAN, ReLUKAN, HRKAN, ChebyshevKAN, RBFKAN, and the original SplineKAN. Across all tasks, FEKAN demonstrates substantially faster convergence and consistently higher approximation accuracy than the underlying baseline architectures. We also establish the theoretical foundations for FEKAN, showing its superior representation capacity compared to KAN, which contributes to improved accuracy and efficiency.

Sidharth S. Menon, Ameya D. Jagtap

Aerospace Engineering Department, Worcester Polytechnic Institute, Worcester, MA 01609, USA.

Function Approximation Kolmogorov-Arnold Networks Neural Network Approximation Theory

Dec 18, 2025

Global universal approximation with Brownian signatures

We establish

L^p

-universal approximation theorems for general path-dependent and non-anticipative functionals on suitable rough path spaces, showing that linear functionals acting on signatures of time-extended rough paths are dense with respect to the

L^p

-distance. To that end, we derive global universal approximation theorems for weighted rough path spaces. We demonstrate that these

L^p

-universal approximation theorems apply to Gaussian processes, in particular, to fractional Brownian motion. As a consequence, linear functionals on the signature of the time-extended Brownian motion can approximate any

p

-integrable stochastic process adapted to the Brownian filtration, including solutions to stochastic differential equations.

Mihriban Ceylan, David J. Prömel

Universal Approximation Path Signatures Function Approximation