Distributionally robust approximation property of neural networks

Mihriban Ceylan; David J. Prömel

stat.MLOct 10, 2025

Distributionally robust approximation property of neural networks

Authors: Mihriban Ceylan, David J. Prömel

Abstract

The universal approximation property uniformly with respect to weakly compact families of measures is established for several classes of neural networks. To that end, we prove that these neural networks are dense in Orlicz spaces, thereby extending classical universal approximation theorems even beyond the traditional $L^p$ -setting. The covered classes of neural networks include widely used architectures like feedforward neural networks with non-polynomial activation functions, deep narrow networks with ReLU activation functions and functional input neural networks.

Neural Network Approximation Theory

Neural network approximation theory studies how network size, depth, activations, and structure determine the accuracy of functions and operators.

See all 242 papers

Approximation Theory for Neural Networks: Old and New
May 20, 2026Soumendu Sundar Mukherjee, Himasish TalukdarUniversal Approximation Neural Network Approximation Theory
Statistics and Mathematics Unit, Indian Statistical Institute, Kolkata
Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation
Jun 25, 2026Anastasis Kratsios, Simone Brugiapaglia, Bum Jun Kim +2Universal Approximation Neural Network Approximation Theory
McMaster University & Vector Institute, Toronto, ON, Canada · Concordia University, Montréal, QC, Canada · University of Tokyo, Tokyo, Japan +2
On shallow feedforward neural networks with inputs from a topological space
Apr 3, 2025Vugar IsmailovFeedforward Neural Networks Neural Network Approximation Theory
Institute of Mathematics and Mechanics, Baku, Azerbaijan · Center for Mathematics and its Applications, Khazar University, Baku, Azerbaijan

May 20, 2026J/K move · Enter open · S save

Approximation Theory for Neural Networks: Old and New

Soumendu Sundar Mukherjee, Himasish Talukdar

Statistics and Mathematics Unit, Indian Statistical Institute, Kolkata

Universal Approximation Neural Network Approximation Theory Feedforward Neural Networks Neural Network Expressivity Function Approximation

Open paper AskPDF arXiv

Universal approximation theorems provide a mathematical explanation for the expressive power of neural networks. They assert that, under mild conditions on the activation function, feedforward neural networks are dense in broad function classes, such as continuous functions on compact subsets of

\mathbb{R}^d

L^p

spaces, or Sobolev spaces. Over the past four decades, these qualitative universality results have evolved into a rich quantitative theory addressing approximation rates, parameter efficiency, and the role of architectural features such as depth and width. This survey presents several glimpses into this theory. We review classical density results for single-hidden-layer networks, as well as quantitative bounds that relate approximation error to network size and smoothness assumptions on target functions. Particular emphasis is placed on depth--width trade-offs and on results demonstrating that deeper architectures can achieve superior parameter efficiency for structured function classes. In addition to standard feedforward neural networks, we also review recent developments on Kolmogorov--Arnold Networks (KANs), which offer an alternative architectural paradigm and whose approximation-theoretic properties have begun to attract significant theoretical attention.

Abstract

Neural Network Approximation Theory

Explore similar work

Approximation Theory for Neural Networks: Old and New

Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation

On shallow feedforward neural networks with inputs from a topological space