Layer ReLU Network

Layer ReLU networks, utilizing the rectified linear unit activation function, are a central focus in neural network research, with current efforts concentrating on understanding their approximation capabilities, optimization challenges, and generalization properties. Investigations explore various architectures, including deep and shallow networks, recurrent networks, and those incorporating techniques like Bayesian methods and quadratic formulations for specific tasks such as combinatorial optimization. These studies aim to improve both the theoretical understanding of ReLU networks and their practical application in diverse fields, ranging from solving partial differential equations to image classification and beyond.

Papers

October 29, 2024

Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient
Vu C. Dinh, Lam Si Tung Ho, Cuong V. Nguyen
Hamiltonian Monte Carlo ReLU Neural Network Layer ReLU Network Novel Single Integrator Bayesian Neural

October 21, 2024

ReLU neural network approximation to piecewise constant functions
Zhiqiang Cai, Junpyo Choi, Min Liu
ReLU Network ReLU Layer Neural Network Approximation Layer ReLU Network Approximation Property Discontinuity Discrimination Constant Measure

October 8, 2024

Polynomial Time Cryptanalytic Extraction of Deep Neural Networks in the Hard-Label Setting
Nicholas Carlini, Jorge Chávez-Saab, Anna Hambitzer, Francisco Rodríguez-Henríquez, Adi Shamir
Deep Neural Network Polynomial Time Layer ReLU Network Deep Neural Network Parameter Hard Label Differential Neural

October 2, 2024

Uncertainty Quantification with Bayesian Higher Order ReLU KANs
James Giroux, Cristiano Fanelli
Uncertainty Quantification Kolmogorov Arnold Network Stochastic Way Aleatoric Uncertainty Bayesian Approach Layer ReLU Network

July 4, 2024

Implicit Hypersurface Approximation Capacity in Deep ReLU Networks
Jonatan Vallin, Karl Larsson, Mats G. Larson
Neural Network ReLU Layer ReLU Activation Deep ReLU Network Approximation Theory Layer ReLU Network Approximation Power

June 27, 2024

Dataless Quadratic Neural Networks for the Maximum Independent Set Problem
Ismail Alkhouri, Cedric Le Denmat, Yingjie Li, Cunxi Yu, Jia Liu, Rongrong Wang, Alvaro Velasquez
Gradient Based Combinatorial Optimization Problem Quadratic Neural Network Layer ReLU Network Independent Set

May 8, 2024

Stability and Performance Analysis of Discrete-Time ReLU Recurrent Neural Networks
Sahel Vahedi Noori, Bin Hu, Geir Dullerud, Peter Seiler
Recurrent Neural Network Core Stability ReLU Network Performance Analysis ReLU Activation Layer ReLU Network Sample Wise Gain

May 3, 2024

Three Quantization Regimes for ReLU Networks
Weigutian Ou, Philipp Schenkel, Helmut Bölcskei
ReLU Network Lipschitz Operator Quantization Level Layer ReLU Network

February 6, 2024

Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Time
Sungyoon Kim, Mert Pilanci
Polynomial Time Convex Relaxation Two Layer ReLU Local Gradient Layer ReLU Network Optimality Gap

October 12, 2023

Polynomial Time Cryptanalytic Extraction of Neural Network Models
Adi Shamir, Isaac Canales-Martinez, Anna Hambitzer, Jorge Chavez-Saab, Francisco Rodrigez-Henriquez, Nitin Satpute
Neural Network Deep Neural Network Black Box Neural Network Model Polynomial Time Hidden Layer Layer ReLU Network

October 11, 2023

Deep ReLU networks and high-order finite element methods II: Chebyshev emulation
Joost A. A. Opschoor, Christoph Schwab
ReLU Network Deep ReLU Network Layer ReLU Network Chebyshev Davidson Algorithm Neural Network Surrogate High Order

July 24, 2023

Efficiently Learning One-Hidden-Layer ReLU Networks via Schur Polynomials
Ilias Diakonikolas, Daniel M. Kane
Tensor Decomposition ReLU Activation Layer ReLU Network Order Moment Moment Tensor

July 1, 2023

Sparsity-aware generalization theory for deep neural networks
Ramchandran Muthukumar, Jeremias Sulam
Deep Neural Network Strong Generalization Sparsity Increase Layer ReLU Network

June 30, 2023

The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks
Mor Shpigel Nacson, Rotem Mulayoff, Greg Ongie, Tomer Michaeli, Daniel Soudry
Implicit Bias ReLU Network Shallow Network Flat Minimum Shallow ReLU Layer ReLU Network

June 10, 2023

Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs
Dmitry Chistikov, Matthias Englert, Ranko Lazic
LeArning Abstract Gradient Flow Single Neuron Level Implicit Bias Teacher Network Shallow ReLU Single Neuron Layer ReLU Network Neural Interpolation

May 25, 2023

Data Topology-Dependent Upper Bounds of Neural Network Widths
Sangmin Lee, Jong Chul Ye
Deep Neural Network Raw Data Universal Approximation Upper Bound Layer ReLU Network Three Layer Neural Network Width Neural Network

March 2, 2023

Penalising the biases in norm regularisation enforces sparsity
Etienne Boursier, Nicolas Flammarion
Regularization Model Sparsity Increase Topic Bias Layer ReLU Network Sparse Solution Sparsity Inducing Minimum Norm

December 23, 2022

An Exact Mapping From ReLU Networks to Spiking Neural Networks
Ana Stanojevic, Stanisław Woźniak, Guillaume Bellec, Giovanni Cherubini, Angeliki Pantazi, Wulfram Gerstner
Spiking Neural Network ReLU Network Deep ReLU Network Layer ReLU Network

June 2, 2022

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs
Etienne Boursier, Loucas Pillaud-Vivien, Nicolas Flammarion
Gradient Descent Post Hoc Orthogonalization Shallow ReLU Layer ReLU Network Square Loss Gradient Flow Dynamic

May 31, 2022

Learning (Very) Simple Generative Models Is Hard
Sitan Chen, Jerry Li, Yuanzhi Li
LeArning Abstract Computational Complexity Hidden Layer Unsupervised Visual Representation Learning Layer ReLU Network Simple Generative Polynomial Bound