Shallow ReLU

Shallow ReLU networks, characterized by a single hidden layer using the Rectified Linear Unit (ReLU) activation function, are a focus of research aiming to understand their approximation capabilities and optimization landscapes. Current investigations explore their performance in various applications, including function approximation, control systems, and image processing, often focusing on the impact of random weight initialization and the development of efficient training algorithms like structure-guided Gauss-Newton methods. These studies are significant because they provide theoretical foundations for understanding the behavior of these simple yet powerful networks, leading to improved training strategies and a deeper understanding of their generalization properties.

Papers

October 18, 2024

Topological obstruction to the training of shallow ReLU neural networks
Marco Nurisso, Pierrick Leroy, Francesco Vaccarino
Training Data Gradient Flow ReLU Activation Neural Scaling Law Shallow ReLU Simple Neural Network

October 7, 2024

Function Gradient Approximation with Random Shallow ReLU Networks with Control Applications
Andrew Lamperski, Siddharth Salapaka
Neural Network Value Function Shallow Network Gradient Approximation Shallow ReLU Smooth Function Control System

August 26, 2024

Training-Free Activation Sparsity in Large Language Models
James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, Ben Athiwaratkun
Large Language Model Activation Sparsity Shallow ReLU Model Sparsity

August 20, 2024

Approximation Rates for Shallow ReLU$^k$ Neural Networks on Sobolev Spaces via the Radon Transform
Tong Mao, Jonathan W. Siegel, Jinchao Xu
Shallow Neural Network ReLU Activation Shallow ReLU \Ell_p$ Norm Approximation Rate Radon Transform Sobolev Norm

June 12, 2024

Blind Image Deblurring with FFT-ReLU Sparsity Prior
Abdul Mohaimen Al Radi, Prothito Shovon Majumder, Md. Mosaddek Khan
Shallow ReLU Blur Kernel Deep Learning Pipeline Blind Image ReLU Function

April 7, 2024

A Structure-Guided Gauss-Newton Method for Shallow ReLU Neural Network
Zhiqiang Cai, Tong Ding, Min Liu, Xinyu Liu, Jianlin Xia
Shallow ReLU Gauss Newton

March 25, 2024

Approximation with Random Shallow ReLU Networks with Applications to Model Reference Adaptive Control
Andrew Lamperski, Tyler Lekang
Neural Network Financial Application Average Approximation Adaptive Control Shallow ReLU Approximation Guarantee Approximation Property Model Reference Adaptive

March 9, 2024

Shallow ReLU neural networks and finite elements
Pengzhan Jin
Neural Network ReLU Network Finite Element Piecewise Linear Shallow ReLU Convex Polytope

February 8, 2024

Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escaping, and Network Embedding
Zhengqing Wu, Berfin Simsek, Francois Ged
Network Programming Loss Landscape Saddle Point ReLU Activation Shallow ReLU Stationary Point

November 10, 2023

Minimum norm interpolation by perceptra: Explicit regularization and implicit bias
Jiyoung Park, Ian Pelakh, Stephan Wojtowytsch
Empirical Risk Minimization Implicit Bias Explicit Regularization Shallow ReLU Decay Regularization Minimum Norm

July 24, 2023

Rates of Approximation by ReLU Shallow Neural Networks
Tong Mao, Ding-Xuan Zhou
Average Approximation Function Approximation Shallow Neural Network Shallow Network Shallow ReLU Universal Rate Uniform Approximation

June 30, 2023

The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks
Mor Shpigel Nacson, Rotem Mulayoff, Greg Ongie, Tomer Michaeli, Daniel Soudry
Implicit Bias ReLU Network Shallow Network Flat Minimum Shallow ReLU Layer ReLU Network

June 16, 2023

Training shallow ReLU networks on noisy data using hinge loss: when do we overfit and is it benign?
Erin George, Michael Murray, William Swartworth, Deanna Needell
Training Data Gradient Descent Binary Classification Noisy Data Model Overfitting Two Layer ReLU Benign Overfitting Shallow ReLU Hinge Loss

June 14, 2023

Nonparametric regression using over-parameterized shallow ReLU neural networks
Yunfei Yang, Ding-Xuan Zhou
Neural Network Shallow Neural Network Wide Neural Network Nonparametric Regression Shallow ReLU Rate Maximization

June 10, 2023

Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs
Dmitry Chistikov, Matthias Englert, Ranko Lazic
LeArning Abstract Gradient Flow Single Neuron Level Implicit Bias Teacher Network Shallow ReLU Single Neuron Layer ReLU Network Neural Interpolation

June 5, 2023

Does a sparse ReLU network training problem always admit an optimum?
Quoc-Tung Le, Elisa Riccietti, Rémi Gribonval
Near Optimal Shallow ReLU Sparsity Pattern

May 31, 2023

On the Expressive Power of Neural Networks
Jan Holstermann
Neural Network Universal Approximation Expressive Power Shallow ReLU Deep Narrow