Approximation Power

Approximation power in neural networks investigates the ability of different architectures to accurately represent complex functions, focusing on understanding the relationship between network structure (depth, width, activation functions), parameter constraints, and approximation error. Current research explores this across various models, including feedforward, recurrent, and convolutional networks, analyzing approximation bounds for specific function classes and developing techniques to improve efficiency, such as optimized activation function approximations. These investigations are crucial for both theoretical understanding of deep learning's capabilities and for practical applications, leading to more efficient and effective algorithms for tasks like regression, classification, and solving partial differential equations.

Papers

September 25, 2024

Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?
Li Liu, Tengchao Yu, Heng Yong
Neural Network Continuum Limit Universal Approximation Norm Bounded Approximation Power Approximation Capability

September 9, 2024

Approximation Bounds for Recurrent Neural Networks with Application to Regression
Yuling Jiao, Yang Wang, Bokai Yan
Application Proficiency Recurrent Neural Network Novel Regression ReLU Network \Sigma}{\Delta}$ Low Pas RNN Approximation Guarantee Approximation Power

July 4, 2024

Implicit Hypersurface Approximation Capacity in Deep ReLU Networks
Jonatan Vallin, Karl Larsson, Mats G. Larson
Neural Network ReLU Layer ReLU Activation Deep ReLU Network Approximation Theory Layer ReLU Network Approximation Power

June 21, 2024

PEANO-ViT: Power-Efficient Approximations of Non-Linearities in Vision Transformers
Mohammad Erfan Sadeghi, Arash Fayyazi, Seyedarmin Azizi, Massoud Pedram
Vision Transformer Field Programmable Gate Array Layer Normalization Divide and Conquer Approximation Power Non Linearity FPGA Resource

April 26, 2024

Multi-layer random features and the approximation power of neural networks
Rustem Takhanov
Neural Network Multi Layer Neural Architecture Kernel Hilbert Space Random Feature Random Field Approximation Power Neural Network Gaussian Process

December 20, 2023

A Minimal Control Family of Dynamical Syetem for Universal Approximation
Yifei Duan, Yongqiang Cai
Dynamical System Optimal Control Universal Approximation Approximation Power Nonlinear Function Linear Mapping

May 26, 2023

On the Generalization and Approximation Capacities of Neural Controlled Differential Equations
Linus Bleistein, Agathe Guilloux
Neural Network Deep Neural Network Strong Generalization Supervised Learning Differential Equation Sampling Bias Approximation Power

May 23, 2023

Basis Function Encoding of Numerical Features in Factorization Machines for Improved Accuracy
Alex Shtoff, Elie Abboud, Rotem Stram, Oren Somekh
Numerical Data Accuracy Improvement Basis Function Factorization Machine Approximation Power

April 10, 2023

Approximation of Nonlinear Functionals Using Deep ReLU Networks
Linhao Song, Jun Fan, Di-Rong Chen, Ding-Xuan Zhou
Average Approximation Rectified Linear Unit Approximation Power Functional Deep Neural Network Statistical Functionals

January 29, 2023

On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Network
Shijun Zhang, Jianfeng Lu, Hongkai Zhao
Expressive Power ReLU Neural Network Approximation Power Continuous Depth Function Composition

January 19, 2023

The necessity of depth for artificial neural networks to approximate certain classes of smooth and bounded functions without the curse of dimensionality
Lukas Gonon, Robin Graeber, Arnulf Jentzen
Neural Network Large Depth Simple Function Data Dimensionality Class Specific Artificial Neural Network Approximation Power

November 23, 2022

Replacing Automatic Differentiation by Sobolev Cubatures fastens Physics Informed Neural Nets and strengthens their Approximation Power
Juan Esteban Suarez Cardona, Michael Hecht
Physic Informed Neural Network Automatic Differentiation Approximation Power Inverse PDE Problem Loss Estimation Variational Loss

September 23, 2022

Achieve the Minimum Width of Neural Networks for Universal Approximation
Yongqiang Cai
Neural Network Universal Approximation Wide Neural Network Universal Approximators \Ell_p$ Norm Minimum Width Approximation Power

July 19, 2022

Approximation Power of Deep Neural Networks: an explanatory mathematical survey
Mohammad Motamed
Deep Neural Network Deep Network Back Propagation Deep ReLU Network Nonlinear Function Approximation Approximation Power

January 24, 2022

Approximation bounds for norm constrained neural networks with applications to regression and GANs
Yuling Jiao, Yang Wang, Yunfei Yang
Neural Network GAN Model Novel Regression Average Approximation Rapid Norm Growth Approximation Power Target Norm

November 15, 2021

Deep Network Approximation in Terms of Intrinsic Parameters
Zuowei Shen, Haizhao Yang, Shijun Zhang
Deep Learning Learnable Parameter Model Parameter Approximation Power Contentious Term Deep Neural Network Approximation

Approximation Power

Papers

Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

Approximation Bounds for Recurrent Neural Networks with Application to Regression

Implicit Hypersurface Approximation Capacity in Deep ReLU Networks

PEANO-ViT: Power-Efficient Approximations of Non-Linearities in Vision Transformers

Multi-layer random features and the approximation power of neural networks

A Minimal Control Family of Dynamical Syetem for Universal Approximation

On the Generalization and Approximation Capacities of Neural Controlled Differential Equations

Basis Function Encoding of Numerical Features in Factorization Machines for Improved Accuracy

Approximation of Nonlinear Functionals Using Deep ReLU Networks

On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Network

The necessity of depth for artificial neural networks to approximate certain classes of smooth and bounded functions without the curse of dimensionality

Replacing Automatic Differentiation by Sobolev Cubatures fastens Physics Informed Neural Nets and strengthens their Approximation Power

Achieve the Minimum Width of Neural Networks for Universal Approximation

Approximation Power of Deep Neural Networks: an explanatory mathematical survey

Approximation bounds for norm constrained neural networks with applications to regression and GANs

Deep Network Approximation in Terms of Intrinsic Parameters