ReLU Network

ReLU networks, a class of neural networks utilizing the rectified linear unit activation function, are a central focus in deep learning research, with current efforts concentrating on understanding their theoretical properties, improving training efficiency, and enhancing their interpretability. Research explores various aspects, including approximation capabilities, generalization behavior (especially concerning benign overfitting), and the impact of network architecture (depth, width, sparsity) on performance. These investigations are crucial for advancing both the theoretical foundations of deep learning and the development of more efficient and reliable machine learning applications across diverse fields.

Papers

May 3, 2024

April 9, 2024

Efficient Quantum Circuits for Machine Learning Activation Functions including Constant T-depth ReLU
Wei Zi, Siyi Wang, Hyunji Kim, Xiaoming Sun, Anupam Chattopadhyay, Patrick Rebentrost
Quantum Machine Learning Activation Function ReLU Network Lookup Table Leaky ReLU Fault Tolerant Quantum Efficient Quantum

April 5, 2024

Half-Space Feature Learning in Neural Networks
Mahesh Lorik Yadav, Harish Guruprasad Ramaswamy, Chandrashekar Lakshminarayanan
Neural Network Feature Learning ReLU Network Deep Linear Learning Halfspaces Gated Linear

March 20, 2024

Multi-agent Reinforcement Traffic Signal Control based on Interpretable Influence Mechanism and Biased ReLU Approximation
Zhiyue Luo, Jun Xu, Fanglin Chen
Multi Agent Reinforcement Learning ReLU Network Traffic Signal Control Traffic Network Multi Agent Actor Critic

March 9, 2024

Shallow ReLU neural networks and finite elements
Pengzhan Jin
Neural Network ReLU Network Finite Element Piecewise Linear Shallow ReLU Convex Polytope

March 2, 2024

A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Features
Emi Zeger, Yifei Wang, Aaron Mishkin, Tolga Ergen, Emmanuel Candès, Mert Pilanci
Deep Neural Network ReLU Network Etruscan Mirror LASSO Regression Deep Narrow Low Dimension

February 13, 2024

January 8, 2024

Universal Consistency of Wide and Deep ReLU Neural Networks and Minimax Optimal Convergence Rates for Kolmogorov-Donoho Optimal Function Classes
Hyunouk Ko, Xiaoming Huo
Neural Network Simple Classifier Convergence Rate ReLU Network Logistics System Neural Network Classifier Kolmogorov Complexity Universal Consistency

December 7, 2023

Reverse Engineering Deep ReLU Networks An Optimization-based Algorithm
Mehrab Hamidi
Convex Optimization ReLU Network Deep ReLU Network L2 Regularization Network Reconstruction

November 30, 2023

The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks
Lénaïc Chizat, Praneeth Netrapalli
Deep Neural Network Feature Learning ReLU Network Hyper Parameter Hierarchical Learning Speed Prediction

November 28, 2023

TransNeXt: Robust Foveal Visual Perception for Vision Transformers
Dai Shi
Vision Transformer Channel Attention ReLU Network Human Visual Perception Attention Aggregation

November 18, 2023

Polynomial-Time Solutions for ReLU Network Training: A Complexity Classification via Max-Cut and Zonotopes
Yifei Wang, Mert Pilanci
Polynomial Time ReLU Network ReLU Layer Two Layer ReLU Polynomial Time Approximation Polynomial Zonotopes Decay Regularization Difficulty Score

November 7, 2023

Preventing Arbitrarily High Confidence on Far-Away Data in Point-Estimated Discriminative Neural Networks
Ahmad Rashid, Serena Hacker, Guojun Zhang, Agustinus Kristiadi, Pascal Poupart
Training Data Raw Data Neural Architecture ReLU Network High Confidence Deterministic Neural Network High Confidence Prediction

November 1, 2023

SmoothHess: ReLU Network Feature Interactions via Stein's Lemma
Max Torop, Aria Masoomi, Davin Hill, Kivanc Kose, Stratis Ioannidis, Jennifer Dy
Second Order ReLU Network Feature Interaction Smoothing Factor Fundamental Lemma

October 28, 2023

The Evolution of the Interplay Between Input Distributions and Linear Regions in Networks
Xuan Qi, Yi Wei
Deep Neural Network Network Programming Deep Network Specie Evolution ReLU Network Convex Set Input Distribution Linear Region

October 26, 2023

Optimization dependent generalization bound for ReLU networks based on sensitivity in the tangent bundle
Dániel Rácz, Mihály Petreczky, András Csertán, Bálint Daróczy
Deep Learning Deep Neural Network Optimization Purpose Gradient Descent Generalization Error ReLU Network Network Sensitivity Tangent Bundle

October 11, 2023

Deep ReLU networks and high-order finite element methods II: Chebyshev emulation
Joost A. A. Opschoor, Christoph Schwab
ReLU Network Deep ReLU Network Layer ReLU Network Chebyshev Davidson Algorithm Neural Network Surrogate High Order

October 5, 2023

The Geometric Structure of Fully-Connected ReLU Layers
Jonatan Vallin, Karl Larsson, Mats G. Larson
Convolutional Neural Network ReLU Network ReLU Layer Geometric Structure Geometric Complexity

ReLU Network

Papers

Three Quantization Regimes for ReLU Networks

Dependency-Aware Semi-Structured Sparsity of GLU Variants in Large Language Models

Efficient Quantum Circuits for Machine Learning Activation Functions including Constant T-depth ReLU

Half-Space Feature Learning in Neural Networks

Multi-agent Reinforcement Traffic Signal Control based on Interpretable Influence Mechanism and Biased ReLU Approximation

Shallow ReLU neural networks and finite elements

A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Features

Depth Separation in Norm-Bounded Infinite-Width Neural Networks

Geometry-induced Implicit Regularization in Deep ReLU Neural Networks

Universal Consistency of Wide and Deep ReLU Neural Networks and Minimax Optimal Convergence Rates for Kolmogorov-Donoho Optimal Function Classes

Reverse Engineering Deep ReLU Networks An Optimization-based Algorithm

The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks

TransNeXt: Robust Foveal Visual Perception for Vision Transformers

Polynomial-Time Solutions for ReLU Network Training: A Complexity Classification via Max-Cut and Zonotopes

Preventing Arbitrarily High Confidence on Far-Away Data in Point-Estimated Discriminative Neural Networks

SmoothHess: ReLU Network Feature Interactions via Stein's Lemma

The Evolution of the Interplay Between Input Distributions and Linear Regions in Networks

Optimization dependent generalization bound for ReLU networks based on sensitivity in the tangent bundle

Deep ReLU networks and high-order finite element methods II: Chebyshev emulation

The Geometric Structure of Fully-Connected ReLU Layers