Universal Approximation
Universal approximation theory explores the ability of neural networks to approximate any continuous function to arbitrary accuracy. Current research focuses on refining approximation bounds for various network architectures (including feedforward, recurrent, and transformer networks), investigating the impact of parameter constraints (e.g., bounded weights, quantization), and extending the theory to encompass broader input spaces (e.g., topological vector spaces, non-metric spaces) and operator learning. These advancements provide a stronger theoretical foundation for deep learning, informing model design, optimization strategies, and ultimately improving the reliability and efficiency of applications across diverse fields.
Papers
Decomposition of Equivariant Maps via Invariant Maps: Application to Universal Approximation under Symmetry
Akiyoshi Sannai, Yuuki Takai, Matthieu Cordonnier
Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?
Li Liu, Tengchao Yu, Heng Yong
Neural Networks Trained by Weight Permutation are Universal Approximators
Yongqiang Cai, Gaohang Chen, Zhonghua Qiao
Universal Approximation Theory: The Basic Theory for Transformer-based Large Language Models
Wei Wang, Qing Li
Expressivity of Neural Networks with Random Weights and Learned Biases
Ezekiel Williams, Avery Hee-Woon Ryoo, Thomas Jiralerspong, Alexandre Payeur, Matthew G. Perich, Luca Mazzucato, Guillaume Lajoie