Leaky ReLU

Leaky ReLU, a variation of the ReLU activation function used in neural networks, is a subject of ongoing research focused on understanding its theoretical properties and practical applications. Current studies explore its role in overcoming the curse of dimensionality when approximating solutions to partial differential equations, analyzing its implicit bias in gradient descent optimization, and developing efficient quantum circuit implementations. These investigations are significant for advancing the theoretical foundations of deep learning and improving the efficiency and performance of neural networks in various applications, including scientific computing and machine learning.

Papers

October 22, 2024

Leaky ReLUs That Differ in Forward and Backward Pass Facilitate Activation Maximization in Deep Neural Networks
Christoph Linse, Erhardt Barth, Thomas Martinetz
Convolutional Neural Network Deep Neural Network Gradient Computation Forward Pas Activation Maximization Leaky ReLU Test Stimulus

October 16, 2024

On the Role of Activation Functions in EEG-To-Text Decoder
Zenon Lamprou, Iakovos Tenedios, Yashar Moshfeghi
Integral Role Functional Magnetic Resonance Imaging Activation Function Polynomial Activation Neural Network Performance Leaky ReLU EEG to Text

September 30, 2024

Multilevel Picard approximations and deep neural networks with ReLU, leaky ReLU, and softplus activation overcome the curse of dimensionality when approximating semilinear parabolic partial differential equations in $L^p$-sense
Ariel Neufeld, Tuan Anh Nguyen
Neural Network Deep Neural Network \Ell_p$ Norm Parabolic Partial Differential Equation Leaky ReLU Kolmogorov Flow

June 16, 2024

Deep neural networks with ReLU, leaky ReLU, and softplus activation provably overcome the curse of dimensionality for space-time solutions of semilinear partial differential equations
Julia Ackermann, Arnulf Jentzen, Benno Kuckuck, Joshua Lee Padgett
Deep Neural Network Partial Differential Equation Data Dimensionality High Dimensional Partial Differential Equation Leaky ReLU

May 27, 2024

Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets
Arthur Jacot, Alexandre Kaiser
Feature Learning Low Dimensional Representation Dimensional Input Fully Connected Neural Network Leaky ReLU Hamiltonian Function Bottleneck Structure

April 9, 2024

Efficient Quantum Circuits for Machine Learning Activation Functions including Constant T-depth ReLU
Wei Zi, Siyi Wang, Hyunji Kim, Xiaoming Sun, Anupam Chattopadhyay, Patrick Rebentrost
Quantum Machine Learning Activation Function ReLU Network Lookup Table Leaky ReLU Fault Tolerant Quantum Efficient Quantum

March 11, 2024

Benign overfitting in leaky ReLU networks with moderate input dimension
Kedar Karhadkar, Erin George, Michael Murray, Guido Montúfar, Deanna Needell
Benign Overfitting Leaky ReLU Input Dimension

February 19, 2024

The effect of Leaky ReLUs on the training and generalization of overparameterized networks
Yinglong Guo, Shaohan Li, Gilad Lerman
Neural Network Strong Generalization Training Data Mixed Effect Network Programming Generalization Error Leaky ReLU

October 29, 2023

Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data
Yiwen Kou, Zixiang Chen, Quanquan Gu
Gradient Descent Gradient Flow Implicit Bias Two Layer ReLU Post Hoc Orthogonalization Leaky ReLU

October 19, 2023

Training Dynamics of Deep Network Linear Regions
Ahmed Imtiaz Humayun, Randall Balestriero, Richard Baraniuk
Deep Network Generalization Performance Training Dynamic Deep Linear Leaky ReLU

September 24, 2023

Deep neural networks with ReLU, leaky ReLU, and softplus activation provably overcome the curse of dimensionality for Kolmogorov partial differential equations with Lipschitz nonlinearities in the $L^p$-sense
Julia Ackermann, Arnulf Jentzen, Thomas Kruse, Benno Kuckuck, Joshua Lee Padgett
Deep Neural Network High Dimensional Partial Differential Equation \Ell_p$ Norm Leaky ReLU Kolmogorov Flow Canonical Lipschitz Mechanism

June 14, 2023

The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks
Aaron Spieler, Nasim Rahaman, Georg Martius, Bernhard Schölkopf, Anna Levina
Neuron Model Long Horizon Task Leaky ReLU Dendritic Tree

May 29, 2023

Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation
Li'ang Li, Yifei Duan, Guanghua Ji, Yongqiang Cai
Universal Approximation Minimum Width Leaky ReLU Continuous Domain

March 2, 2023

Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization
Spencer Frei, Gal Vardi, Peter L. Bartlett, Nathan Srebro
Linear Classifier Margin Maximization Benign Overfitting Class Conditional Leaky ReLU

October 13, 2022

Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data
Spencer Frei, Gal Vardi, Peter L. Bartlett, Nathan Srebro, Wei Hu
Gradient Descent Low Rank Gradient Flow High Dimensional Data Implicit Bias Fully Connected Neural Network Homogeneous Neural Network Leaky ReLU

May 13, 2022

Convergence of Deep Neural Networks with General Activation Functions and Pooling
Wentao Huang, Yuesheng Xu, Haizhang Zhang
Deep Learning Deep Neural Network Deep Convolutional Neural Network Early Stage Convergence Activation Function Rectified Linear Unit Leaky ReLU

May 11, 2022

An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsity
Wei Liu, Xin Liu, Xiaojun Chen
Lagrangian Relaxation Nonlinear Constraint Leaky ReLU Nonsmooth Nonconvex Optimization Group Sparsity Group Sparse

Leaky ReLU

Papers

Leaky ReLUs That Differ in Forward and Backward Pass Facilitate Activation Maximization in Deep Neural Networks

On the Role of Activation Functions in EEG-To-Text Decoder

Multilevel Picard approximations and deep neural networks with ReLU, leaky ReLU, and softplus activation overcome the curse of dimensionality when approximating semilinear parabolic partial differential equations in $L^p$-sense

Deep neural networks with ReLU, leaky ReLU, and softplus activation provably overcome the curse of dimensionality for space-time solutions of semilinear partial differential equations

Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets

Efficient Quantum Circuits for Machine Learning Activation Functions including Constant T-depth ReLU

Benign overfitting in leaky ReLU networks with moderate input dimension

The effect of Leaky ReLUs on the training and generalization of overparameterized networks

Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data

Training Dynamics of Deep Network Linear Regions

Deep neural networks with ReLU, leaky ReLU, and softplus activation provably overcome the curse of dimensionality for Kolmogorov partial differential equations with Lipschitz nonlinearities in the $L^p$-sense

The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks

Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation

Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization

Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data

Convergence of Deep Neural Networks with General Activation Functions and Pooling

An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsity