ReLU Activation

The ReLU (Rectified Linear Unit) activation function, a simple yet powerful element in neural networks, is a central focus of ongoing research aimed at understanding its properties and improving its application. Current research explores ReLU's role in approximation theory, examining its capacity to represent functions of varying complexity within different model architectures, including deep and shallow networks, transformers, and recurrent neural networks. This research is crucial for advancing both theoretical understanding of neural network behavior and practical applications, particularly in improving training efficiency, enhancing model robustness, and optimizing inference speed in large-scale models like LLMs. Furthermore, investigations into ReLU's interaction with other components, such as batch normalization and various optimization algorithms, are actively pursued to address challenges like gradient explosion and improve overall model performance.

Papers

November 29, 2023

Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training
Max Milkert, David Hyde, Forrest Laine
Training Data Large Depth ReLU Activation Piecewise Linear Model Weight Activation Pattern Weighted Network Neural Network Initialization Exponential Growth

November 7, 2023

Improved weight initialization for deep and narrow feedforward neural network
Hyunwoo Lee, Yunho Kim, Seung Yeop Yang, Hayoung Choi
Neural Network Deep Neural Network ReLU Activation Weight Matrix Weight Initialization Robust Initialization

October 6, 2023

ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models
Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, Mehrdad Farajtabar
LLM Inference ReLU Layer ReLU Activation Activation Sparsity Inference Memory Usage

August 10, 2023

Optimizing Performance of Feedforward and Convolutional Neural Networks through Dynamic Activation Functions
Chinmay Rane, Kanishka Tyagi, Michael Manry
Convolutional Neural Network ReLU Activation Performance Optimization Transformer Feed Forward Layer Piecewise Polynomial

July 24, 2023

June 30, 2023

Efficient uniform approximation using Random Vector Functional Link networks
Palina Salanevich, Olov Schavemaker
Hidden Layer ReLU Activation Random Weight Random Vector Functional Link Uniform Approximation

June 20, 2023

Principles for Initialization and Architecture Selection in Graph Neural Networks with ReLU Activations
Gage DeZoort, Boris Hanin
Graph Neural Network General Principle New Initialization ReLU Activation Residual Connection

May 25, 2023

Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks
Wenlin Chen, Hong Ge
Neural Network ReLU Network ReLU Activation ReLU Neural Network Area Preserving Parameterization

May 2, 2023

Hamming Similarity and Graph Laplacians for Class Partitioning and Adversarial Image Detection
Huma Jamil, Yajing Liu, Turgay Caglar, Christina M. Cole, Nathaniel Blanchard, Christopher Peterson, Michael Kirby
Adversarial Image ReLU Activation Graph Laplacian Adversarial Detection Neural Network Representation Class Separation Hamming Space

April 23, 2023

The Disharmony between BN and ReLU Causes Gradient Explosion, but is Offset by the Correlation between Activations
Inyoung Paik, Jaesik Choi
Learning Rate Total Correlation Targeted Activation Penalty ReLU Layer ReLU Activation Training Instability

April 20, 2023

Learning Narrow One-Hidden-Layer ReLU Networks
Sitan Chen, Zehao Dou, Surbhi Goel, Adam R Klivans, Raghu Meka
Polynomial Time ReLU Activation Individual Neuron Deep Narrow Moment Tensor

April 19, 2023

Points of non-linearity of functions generated by random neural networks
David Holmes
Neural Network Exceptional Point Simple Function ReLU Activation Random Neural Network Non Linearity Information Complexity

April 6, 2023

Training a Two Layer ReLU Network Analytically
Adrian Barbu
Gradient Descent Two Layer Neural Network Adam Optimizer ReLU Activation Two Layer ReLU

April 5, 2023

Hybrid Zonotopes Exactly Represent ReLU Neural Networks
Joshua Ortiz, Alyssa Vellucci, Justin Koeln, Justin Ruths
Neural Network MNIST Dataset ReLU Activation Polynomial Zonotopes

March 29, 2023

An Over-parameterized Exponential Regression
Yeqi Gao, Sridhar Mahadevan, Zhao Song
Neural Network Neural Process ReLU Activation Parameterized Model Exponential Activation Function

March 6, 2023

On the existence of optimal shallow feedforward networks with ReLU activation
Steffen Dereich, Sebastian Kassing
ReLU Network ReLU Activation Global Minimum Local Minimizers

October 28, 2022

Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions
Stanislas Ducotterd, Alexis Goujon, Pakshal Bohra, Dimitris Perdios, Sebastian Neumayer, Michael Unser
Neural Architecture Activation Function ReLU Activation Total Variation Lipschitz Network Spline Approximation Functional Constrained Optimization

August 4, 2022

Agnostic Learning of General ReLU Activation Using Gradient Descent
Pranjal Awasthi, Alex Tang, Aravindan Vijayaraghavan
Gradient Descent ReLU Activation Agnostic Learning Neural Network Output ReLU Function ReLU Operation

July 15, 2022

The Mechanical Neural Network(MNN) -- A physical implementation of a multilayer perceptron for education and hands-on experimentation
Axel Schaffland
Neural Network Multi Layer Education Domain Single Neuron Level Multi Layer Perceptron ReLU Activation Hardware Implementation Learning Hand on Electronics