Approximation Rate

Approximation rate research focuses on determining how efficiently different models, such as deep neural networks (including ReLU, convolutional, and transformer architectures) and radial basis functions, can approximate complex functions. Current studies analyze approximation rates across various function spaces (e.g., Sobolev, Besov) using different error metrics and explore the impact of model parameters (depth, width, number of experts) on accuracy. These investigations are crucial for understanding the capabilities and limitations of different machine learning models, informing the design of more efficient algorithms, and improving the accuracy and reliability of applications in scientific computing, signal processing, and other fields.

Papers

November 6, 2024

Weighted Sobolev Approximation Rates for Neural Networks on Unbounded Domains
Ahmed Abdeljawad, Thomas Dittrich
Neural Network Domain Name Shallow Neural Network Shallow Network Approximation Rate Sobolev Norm Barron Space

October 2, 2024

On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding
Kevin Xu, Issei Sato
Feature Enhancement Theoretical Analysis Function Approximation Expressive Power Approximation Rate Temporal Encoding

September 2, 2024

On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks
Yunfei Yang
Neural Network Average Approximation Function Space Approximation Rate Sparse Vector Sobolev Space

August 20, 2024

Approximation Rates for Shallow ReLU$^k$ Neural Networks on Sobolev Spaces via the Radon Transform
Tong Mao, Jonathan W. Siegel, Jinchao Xu
Shallow Neural Network ReLU Activation Shallow ReLU \Ell_p$ Norm Approximation Rate Radon Transform Sobolev Norm

February 5, 2024

Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts
Anastasis Kratsios, Haitz Sáez de Ocáriz Borde, Takashi Furuya, Marc T. Law
Deep Learning Model Expert Knowledge Mixture of Expert ReLU Layer Sparse Activation Traditional Deep Learning Approximation Rate

February 1, 2024

Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling
Mingze Wang, Weinan E
Transformer Based Sequence Modeling Functional Mechanism Expressive Power Feed Forward Layer Approximation Rate Dot Product Self Attention

December 12, 2023

Towards Optimal Sobolev Norm Rates for the Vector-Valued Regularized Least-Squares Algorithm
Zhu Li, Dimitri Meunier, Mattes Mollenhauer, Arthur Gretton
Reproducing Kernel Hilbert Space Approximation Rate

December 5, 2023

Algorithms for mean-field variational inference via polyhedral optimization in the Wasserstein space
Yiheng Jiang, Sinho Chewi, Aram-Alexandre Pooladian
Practical Algorithm Mean Field Wasserstein Space Approximation Rate Polyhedral Compiler Complexity Change Polyhedral Set

November 8, 2023

Optimal Deep Neural Network Approximation for Korobov Functions with respect to Sobolev Norms
Yahong Yang, Yulong Lu
Deep Neural Network Average Approximation Simple Function Approximation Rate Approximation Property Deep Neural Network Approximation Sobolev Norm Function Approximator

August 10, 2023

On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem
Juncai He
Application Proficiency Average Approximation Expressive Power Piecewise Linear First Order Superposition Approximation Rate

August 7, 2023

Optimal Approximation and Learning Rates for Deep Convolutional Neural Networks
Shao-Bo Lin
Deep Convolutional Neural Network Learning Rate Average Approximation Approximation Rate Pooling Operator

July 28, 2023

Optimal Approximation of Zonoids and Uniform Approximation by Shallow Neural Networks
Jonathan W. Siegel
Shallow Neural Network Shallow ReLU Approximation Rate Hausdorff Distance Uniform Approximation

May 29, 2023

April 4, 2023

Optimal rates of approximation by shallow ReLU$^k$ neural networks and applications to nonparametric regression
Yunfei Yang, Ding-Xuan Zhou
Neural Network Average Approximation Gradient Norm Shallow Neural Network Nonparametric Regression Optimal Rate Shallow ReLU Approximation Rate

February 2, 2023

Sharp Lower Bounds on Interpolation by Deep ReLU Neural Networks at Irregularly Spaced Data
Jonathan W. Siegel
Lower Bound ReLU Network Interpolation Regime Deep ReLU Network Approximation Rate Space Domain

January 30, 2023

Optimal Approximation Complexity of High-Dimensional Functions with Neural Networks
Vincent P. H. Goverse, Jad Hamdan, Jared Tanner
Neural Network High Dimensional Deep ReLU Network Approximation Rate Nonlinear Function Approximation Dimensional Subspace

November 25, 2022

Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces
Jonathan W. Siegel
ReLU Network Function Space Deep ReLU Network \Ell_p$ Norm Approximation Rate Sharp Bound Sobolev Space

July 11, 2022

Neural and spectral operator surrogates: unified construction and expression rate bounds
Lukas Herrmann, Christoph Schwab, Jakob Zech
Neural Topic Construction Industry Approximation Rate Deep Surrogate Operator Surrogate Polynomial Chaos Rate Bound

March 25, 2022

On efficient algorithms for computing near-best polynomial approximations to high-dimensional, Hilbert-valued functions from limited samples
Ben Adcock, Simone Brugiapaglia, Nick Dexter, Sebastian Moraga
Compressed Sensing Efficient Algorithm Approximation Rate Polynomial Approximation Limited Sample Sparse Function Hilbert Transform \Ell_1$ Minimization

Approximation Rate

Papers

Weighted Sobolev Approximation Rates for Neural Networks on Unbounded Domains

On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding

On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks

Approximation Rates for Shallow ReLU$^k$ Neural Networks on Sobolev Spaces via the Radon Transform

Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts

Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling

Towards Optimal Sobolev Norm Rates for the Vector-Valued Regularized Least-Squares Algorithm

Algorithms for mean-field variational inference via polyhedral optimization in the Wasserstein space

Optimal Deep Neural Network Approximation for Korobov Functions with respect to Sobolev Norms

On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem

Optimal Approximation and Learning Rates for Deep Convolutional Neural Networks

Optimal Approximation of Zonoids and Uniform Approximation by Shallow Neural Networks

Forward and Inverse Approximation Theory for Linear Temporal Convolutional Networks

Approximation Rate of the Transformer Architecture for Sequence Modeling

Optimal rates of approximation by shallow ReLU$^k$ neural networks and applications to nonparametric regression

Sharp Lower Bounds on Interpolation by Deep ReLU Neural Networks at Irregularly Spaced Data

Optimal Approximation Complexity of High-Dimensional Functions with Neural Networks

Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces

Neural and spectral operator surrogates: unified construction and expression rate bounds

On efficient algorithms for computing near-best polynomial approximations to high-dimensional, Hilbert-valued functions from limited samples