Nonlinear Activation

Nonlinear activation functions are crucial components of neural networks, enabling them to learn complex, non-linear relationships in data. Current research focuses on understanding the role of these activations in network dynamics, exploring techniques to induce sparsity for efficiency, and investigating the impact of activation linearity on model performance and fairness. This research is significant because it improves our fundamental understanding of neural network behavior, leading to more efficient, robust, and equitable models with applications across diverse fields like computer vision and natural language processing.

Papers

October 26, 2024

Emergence of Globally Attracting Fixed Points in Deep Neural Networks With Nonlinear Activations
Amir Joudaki, Thomas Hofmann
Deep Neural Network World Event Path Breaking Emergence Fixed Point Similar Representation Nonlinear Activation Continuous Kernel

September 2, 2024

Representing Neural Network Layers as Linear Operations via Koopman Operator Theory
Nishant Suresh Aswani, Saif Eddin Jabari, Muhammad Shafique
Dynamical System Koopman Operator Dynamic Mode Decomposition Nonlinear Activation Neural Network Layer Simple Neural Network

June 26, 2024

Learning Neural Networks with Sparse Activations
Pranjal Awasthi, Nishanth Dikkala, Pritish Kamath, Raghu Meka
Neural Network Sparsity Increase Activation Sparsity Sparse Activation Free Counterpart PAC Learnability Nonlinear Activation

May 23, 2024

Mixture of Experts Meets Prompt-Based Continual Learning
Minh Le, An Nguyen, Huy Nguyen, Trang Nguyen, Trang Pham, Linh Van Ngo, Nhat Ho
Continual LEArning Continual Learning Mixture Component Expert Knowledge Nonlinear Activation Continual Learning Performance Prompt Based Continual Learning

February 25, 2024

Deep Neural Network Initialization with Sparsity Inducing Activations
Ilan Price, Nicholas Daultry Ball, Samuel C. H. Lam, Adam C. Jones, Jared Tanner
Activation Sparsity Sparse Activation Nonlinear Activation Neural Network Initialization

February 9, 2024

Low-Rank Learning by Design: the Role of Network Architecture and Activation Linearity in Gradient Rank Collapse
Bradley T. Baker, Barak A. Pearlmutter, Robyn Miller, Vince D. Calhoun, Sergey M. Plis
LeArning Abstract Deep Neural Network Product Design Neural Collapse DNN Architecture Low Rank Structure Network Architecture Nonlinear Activation Rank Collapse

February 6, 2024

December 19, 2023

Improving the Expressive Power of Deep Neural Networks through Integral Activation Transform
Zezhong Zhang, Feng Bao, Guannan Zhang
Neural Network Deep Neural Network Expressive Power ReLU Activation Activation Pattern Nonlinear Activation

December 11, 2023

Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context
Xiang Cheng, Yuxin Chen, Suvrit Sra
Context Learning Gradient Descent Context Information Neural Architecture Nonlinear Activation Dual Network Nonlinear Function

October 25, 2023

Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding
Noam Levi, Alon Beck, Yohai Bar-Sinai
Generalization Performance Deeper Network Linear Estimator Grokking Phenomenon Linear Network Nonlinear Activation

October 1, 2023

JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention
Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, Simon Du
Human Attention Self Attention Layer Layer Transformer Single Scene Specific MLP Nonlinear Activation Hierarchical Generative Multi Joint Joint Cross Attention

September 23, 2023

State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory
Shida Wang, Beichen Xue
State Space Model Memory Trace Sequential Pattern Nonlinear Activation Universal Approximators Sequence Relationship

August 16, 2023

Membrane Potential Batch Normalization for Spiking Neural Networks
Yufei Guo, Yuhan Zhang, Yuanpei Chen, Weihang Peng, Xiaode Liu, Liwen Zhang, Xuhui Huang, Zhe Ma
Neural Network Convolution Layer Neuromorphic Datasets Nonlinear Activation Conventional Neural Network Based Normalization

June 18, 2023

Learn to Enhance the Negative Information in Convolutional Neural Network
Zhicheng Cai, Chenglei Peng, Qiu Shen
Convolutional Neural Network CNN Model Convolutional Neural Network Model Nonlinear Activation Negative Information

May 20, 2023

A Framework for Provably Stable and Consistent Training of Deep Feedforward Networks
Arunselvan Ramaswamy, Shalabh Bhatnagar, Naman Saxena
Reinforcement Learning Neural Network New Framework Gradient Descent Deep Q Learning Consistent Conditioning Nonlinear Activation

May 15, 2023

Accelerated Algorithms for Nonlinear Matrix Decomposition with the ReLU function
Giovanni Seraghiti, Atharva Awari, Arnaud Vandaele, Margherita Porcelli, Nicolas Gillis
Nonlinear Activation Nonlinear Function ReLU Function Accelerated Algorithm Nonlinear Matrix Decomposition

October 11, 2022

On Scrambling Phenomena for Randomly Initialized Recurrent Networks
Vaggos Chatziafratis, Ioannis Panageas, Clayton Sanford, Stelios Andrew Stavroulakis
Recurrent Neural Network Chaotic System Chaotic Behavior Nonlinear Activation Randomly Initialized

October 3, 2022

Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks
Andrea Ceni
Neural Network Deep Neural Network Multi Layer Back Propagation Natural Gradient Random Participation Nonlinear Activation

July 27, 2022

Image sensing with multilayer, nonlinear optical neural networks
Tianyu Wang, Mandar M. Sohoni, Logan G. Wright, Martin M. Stein, Shi-Yuan Ma, Tatsuhiro Onodera, Maxwell G. Anderson, Peter L. McMahon
Neural Network Multi Layer Visual Sensor Optical Imaging Optical Neural Network Nonlinear Activation

Nonlinear Activation

Papers

Emergence of Globally Attracting Fixed Points in Deep Neural Networks With Nonlinear Activations

Representing Neural Network Layers as Linear Operations via Koopman Operator Theory

Learning Neural Networks with Sparse Activations

Mixture of Experts Meets Prompt-Based Continual Learning

Deep Neural Network Initialization with Sparsity Inducing Activations

Low-Rank Learning by Design: the Role of Network Architecture and Activation Linearity in Gradient Rank Collapse

Neural Rank Collapse: Weight Decay and Small Within-Class Variability Yield Low-Rank Bias

Disparate Impact on Group Accuracy of Linearization for Private Inference

Improving the Expressive Power of Deep Neural Networks through Integral Activation Transform

Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context

Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding

JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention

State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory

Membrane Potential Batch Normalization for Spiking Neural Networks

Learn to Enhance the Negative Information in Convolutional Neural Network

A Framework for Provably Stable and Consistent Training of Deep Feedforward Networks

Accelerated Algorithms for Nonlinear Matrix Decomposition with the ReLU function

On Scrambling Phenomena for Randomly Initialized Recurrent Networks

Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks

Image sensing with multilayer, nonlinear optical neural networks