Shallow Neural Network

Shallow neural networks, characterized by a single hidden layer, are a focus of research aiming to understand fundamental aspects of neural network learning and approximation capabilities. Current research investigates their optimization dynamics, exploring various training algorithms like gradient flow and its variants, and analyzing the impact of network architecture (e.g., activation functions, width) and data properties on performance and generalization. This research is significant because it provides crucial insights into the theoretical underpinnings of deep learning, informing the design of more efficient and robust models for applications ranging from function approximation to signal processing and time series analysis.

Papers

February 3, 2024

Analyzing Neural Network-Based Generative Diffusion Models through Convex Optimization
Fangzhao Zhang, Mert Pilanci
Diffusion Model Convex Optimization Neural Topic Generative Diffusion Model Score Based Diffusion Model Shallow Neural Network

December 13, 2023

Space-Time Approximation with Shallow Neural Networks in Fourier Lebesgue spaces
Ahmed Abdeljawad, Thomas Dittrich
Shallow Neural Network Dimensional Approximation Sobolev Norm Barron Space

November 10, 2023

Time Scale Network: A Shallow Neural Network For Time Series Data
Trevor Meyer, Camden Shultz, Najim Dehak, Laureano Moro-Velazquez, Pedro Irazoqui
Multi Scale Time Series Data Shallow Neural Network Time Scale Seizure Prediction

November 2, 2023

Upper and lower bounds for the Lipschitz constant of random neural networks
Paul Geuchen, Thomas Heindl, Dominik Stöger, Felix Voigtlaender
Adversarial Example Adversarial Perturbation Lower Bound Shallow Neural Network Lipschitz Constant Random Neural Network Worst Case Robustness

October 9, 2023

Aggregated f-average Neural Network applied to Few-Shot Class Incremental Learning
Mathieu Vu, Emilie Chouzenoux, Ismail Ben Ayed, Jean-Christophe Pesquet
Model Ensembling Shallow Neural Network Weak Learner Ensemble Machine Learning Weak Learning GNN Aggregation

October 5, 2023

Function-Space Optimality of Neural Architectures With Multivariate Nonlinearities
Rahul Parhi, Michael Unser
Neural Architecture Shallow Neural Network Kernel Banach Space Multivariate System Space Optimality

September 19, 2023

Geometric structure of shallow neural networks and constructive ${\mathcal L}^2$ cost minimization
Thomas Chen, Patricia Muñoz Ewald
Gradient Descent Shallow Neural Network Local Minimum Geometric Structure

July 28, 2023

Optimal Approximation of Zonoids and Uniform Approximation by Shallow Neural Networks
Jonathan W. Siegel
Shallow Neural Network Shallow ReLU Approximation Rate Hausdorff Distance Uniform Approximation

July 25, 2023

Piecewise Linear Functions Representable with Infinite Width Shallow ReLU Neural Networks
Sarah McCarty
Shallow Neural Network Piecewise Linear Rectified Linear Unit

July 24, 2023

Rates of Approximation by ReLU Shallow Neural Networks
Tong Mao, Ding-Xuan Zhou
Average Approximation Function Approximation Shallow Neural Network Shallow Network Shallow ReLU Universal Rate Uniform Approximation

July 11, 2023

Fundamental limits of overparametrized shallow neural networks for supervised learning
Francesco Camilli, Daria Tieplova, Jean Barbier
Neural Network Supervised Learning Neural Network Model Information Theoretic Two Layer Neural Network Shallow Neural Network Fundamental Limit Optimal Generalization

June 14, 2023

Nonparametric regression using over-parameterized shallow ReLU neural networks
Yunfei Yang, Ding-Xuan Zhou
Neural Network Shallow Neural Network Wide Neural Network Nonparametric Regression Shallow ReLU Rate Maximization

May 25, 2023

Embeddings between Barron spaces with higher order activation functions
Tjeerd Jan Heeringa, Len Spek, Felix Schwenninger, Christoph Brune
Jina Embeddings Activation Function Shallow Neural Network Barron Space

May 22, 2023

The Integrated Forward-Forward Algorithm: Integrating Forward-Forward and Shallow Backpropagation With Local Losses
Desmond Y. M. Tang
Back Propagation Neural Process Shallow Neural Network Forward Forward Algorithm Local Loss

April 19, 2023

Leveraging the two timescale regime to demonstrate convergence of neural networks
Pierre Marion, Raphaël Berthier
Neural Network Stochastic Gradient Descent Early Stage Convergence Gradient Flow Neural Tangent Kernel Shallow Neural Network Timescale Analysis Two Timescale

April 4, 2023

Optimal rates of approximation by shallow ReLU$^k$ neural networks and applications to nonparametric regression
Yunfei Yang, Ding-Xuan Zhou
Neural Network Average Approximation Gradient Norm Shallow Neural Network Nonparametric Regression Optimal Rate Shallow ReLU Approximation Rate

February 18, 2023

Generalization and Stability of Interpolating Neural Networks with Minimal Width
Hossein Taheri, Christos Thrampoulidis
Neural Network Strong Generalization Gradient Descent Core Stability Learning Rate Generalization Bound Shallow Neural Network Minimum Width

February 2, 2023

Over-parameterised Shallow Neural Networks with Asymmetrical Node Scaling: Global Convergence Guarantees and Feature Learning
Francois Caron, Fadhel Ayed, Paul Jung, Hoil Lee, Juho Lee, Hongseok Yang
Neural Network Gradient Flow Neural Tangent Kernel Feature Learning Shallow Neural Network Global Convergence Guarantee

February 1, 2023

Multi-Grade Deep Learning
Yuesheng Xu
Deep Learning Shallow Neural Network

December 5, 2022

Improved Convergence Guarantees for Shallow Neural Networks
Alexander Razborov
Neural Network Gradient Descent Convergence Guarantee Shallow Neural Network Quadratic Loss

Shallow Neural Network

Papers

Analyzing Neural Network-Based Generative Diffusion Models through Convex Optimization

Space-Time Approximation with Shallow Neural Networks in Fourier Lebesgue spaces

Time Scale Network: A Shallow Neural Network For Time Series Data

Upper and lower bounds for the Lipschitz constant of random neural networks

Aggregated f-average Neural Network applied to Few-Shot Class Incremental Learning

Function-Space Optimality of Neural Architectures With Multivariate Nonlinearities

Geometric structure of shallow neural networks and constructive ${\mathcal L}^2$ cost minimization

Optimal Approximation of Zonoids and Uniform Approximation by Shallow Neural Networks

Piecewise Linear Functions Representable with Infinite Width Shallow ReLU Neural Networks

Rates of Approximation by ReLU Shallow Neural Networks

Fundamental limits of overparametrized shallow neural networks for supervised learning

Nonparametric regression using over-parameterized shallow ReLU neural networks

Embeddings between Barron spaces with higher order activation functions

The Integrated Forward-Forward Algorithm: Integrating Forward-Forward and Shallow Backpropagation With Local Losses

Leveraging the two timescale regime to demonstrate convergence of neural networks

Optimal rates of approximation by shallow ReLU$^k$ neural networks and applications to nonparametric regression

Generalization and Stability of Interpolating Neural Networks with Minimal Width

Over-parameterised Shallow Neural Networks with Asymmetrical Node Scaling: Global Convergence Guarantees and Feature Learning

Multi-Grade Deep Learning

Improved Convergence Guarantees for Shallow Neural Networks