ReLU Function

The rectified linear unit (ReLU) function, a simple yet powerful activation function in neural networks, is a subject of ongoing research focused on understanding its properties and improving its application. Current research investigates ReLU's impact on network injectivity, its role in various algorithms (e.g., for blind image deblurring and stochastic bandits), and its optimization for efficient and secure inference, including methods to reduce the number of ReLU operations. These efforts aim to enhance the theoretical understanding of ReLU networks and improve their performance in diverse applications, from image processing to machine learning algorithms.

Papers

June 22, 2024

Injectivity of ReLU-layers: Perspectives from Frame Theory
Peter Balazs, Martin Ehler, Daniel Haider
Synthesized View ReLU Layer Frame Consistency ReLU Function

June 12, 2024

Blind Image Deblurring with FFT-ReLU Sparsity Prior
Abdul Mohaimen Al Radi, Prothito Shovon Majumder, Md. Mosaddek Khan
Shallow ReLU Blur Kernel Deep Learning Pipeline Blind Image ReLU Function

May 12, 2024

Stochastic Bandits with ReLU Neural Networks
Kan Xu, Hamsa Bastani, Surbhi Goel, Osbert Bastani
Linear Bandit Stochastic Bandit ReLU Neural Network Regret Rate ReLU Function

October 2, 2023

A path-norm toolkit for modern networks: consequences, promises and challenges
Antoine Gonon, Nicolas Brisebarre, Elisa Riccietti, Rémi Gribonval
Technical Challenge Network Programming Promise Experiential Consequence ReLU Function Norm Based Generalization Bound Path Norm

May 31, 2023

Optimal Sets and Solution Paths of ReLU Networks
Aaron Mishkin, Mert Pilanci
Solution Path ReLU Network Convex Program ReLU Function Optimal Set

May 15, 2023

Accelerated Algorithms for Nonlinear Matrix Decomposition with the ReLU function
Giovanni Seraghiti, Atharva Awari, Arnaud Vandaele, Margherita Porcelli, Nicolas Gillis
Nonlinear Activation Nonlinear Function ReLU Function Accelerated Algorithm Nonlinear Matrix Decomposition

April 26, 2023

Making Models Shallow Again: Jointly Learning to Reduce Non-Linearity and Depth for Latency-Efficient Private Inference
Souvik Kundu, Yuke Zhang, Dake Chen, Peter A. Beerel
Large Depth Convolution Layer Convolutional Block ReLU Layer Private Inference Shallow Layer Non Linearity ReLU Function

April 20, 2023

Securing Neural Networks with Knapsack Optimization
Yakir Gorski, Amir Jevnisek, Shai Avidan
Supervised ImageNet Knapsack Problem Party Computation ReLU Function Secure Deep

March 28, 2023

Bayesian Free Energy of Deep ReLU Neural Network in Overparametrized Cases
Shuya Nagayasu, Sumio Watanabe
Case Relevance Generalization Error Free Energy Statistical Complexity ReLU Function

February 20, 2023

Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initialization
Cameron Jakub, Mihai Nica
Neural Network Deep Neural Network ReLU Network New Initialization Tilting Angle Combinatorial Approach Model Degeneracy ReLU Function

January 23, 2023

Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inference
Souvik Kundu, Shunlin Lu, Yuke Zhang, Jacqueline Liu, Peter A. Beerel
Neural Network LeArning Abstract Deep Neural Network ReLU Layer Secure Approach Private Inference ReLU Function

August 4, 2022

Agnostic Learning of General ReLU Activation Using Gradient Descent
Pranjal Awasthi, Alex Tang, Aravindan Vijayaraghavan
Gradient Descent ReLU Activation Agnostic Learning Neural Network Output ReLU Function ReLU Operation