Differentiable Activation Function

Differentiable activation functions are crucial components of neural networks, enabling efficient gradient-based training through backpropagation. Current research focuses on understanding their impact on model performance, particularly concerning issues like dead neurons and Hessian matrix properties, and exploring alternatives to backpropagation for training networks with non-differentiable activations. This research is significant because it addresses limitations of standard activation functions, leading to improved model robustness, efficiency, and interpretability in various applications, including reinforcement learning and hardware implementations. Furthermore, investigations into differentiable approximations of non-differentiable functions are expanding the range of applicable neural network architectures.

Papers

June 13, 2024

Hadamard Representations: Augmenting Hyperbolic Tangents in RL
Jacob E. Kooi, Mark Hoogendoorn, Vincent François-Lavet
Neural Network Architecture Fast Learning Human Like RL Differentiable Activation Function

June 6, 2024

Provable Bounds on the Hessian of Neural Networks: Derivative-Preserving Reachability Analysis
Sina Sharifi, Mahyar Fazlyab
Neural Network Hessian Matrix Reachability Analysis Error Bound Taylor Expansion Differentiable Activation Function Quantitative Bound

April 9, 2024

A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on $n$-dimensional Classes
Alessandro Benfenati, Alessio Marta
Neural Network Deep Neural Network Random Walk Riemannian Geometry Differentiable Neural Riemannian Geometry Approach Recursive Neural Network Differentiable Activation Function

February 17, 2024

Random Projection Neural Networks of Best Approximation: Convergence theory and practical applications
Gianluca Fabiani
Neural Network Average Approximation Function Approximation Practical Application Random Projection Differentiable Activation Function Convergence Theory

February 4, 2024

Leveraging Continuously Differentiable Activation Functions for Learning in Quantized Noisy Environments
Vivswan Shah, Nathan Youngblood
LeArning Abstract Deep Learning Model Quantization Error Noisy Sample Differentiable Activation Function Analog Domain

July 14, 2023

STCSNN: High energy efficiency spike-train level spiking neural networks with spatio-temporal conversion
Changqing Xu, Yi Liu, Yintang Yang
Neural Network Energy Efficiency Spike Train Neuromorphic Datasets Neuronal Network Spatio Temporal Transformer Differentiable Activation Function

May 12, 2023

$\partial\mathbb{B}$ nets: learning discrete functions by gradient descent
Ian Wright
Gradient Descent Differentiable Neural Differentiable Activation Function Discrete Function \Mathcal{I}$ Net

May 1, 2023

Interpreting Deep Forest through Feature Contribution and MDI Feature Importance
Yi-Xiao He, Shen-Huan Lyu, Yuan Jiang
Random Forest Feature Importance Deep Layer Deep Forest Differentiable Activation Function Deep Regression Forest

June 3, 2022

A Robust Backpropagation-Free Framework for Images
Timothy Zee, Alexander G. Ororbia, Ankur Mali, Ifeoma Nwogu
Back Propagation Deep Learning Algorithm Backward Propagation Differentiable Activation Function System Neuroscience

May 31, 2022

A comparative study of back propagation and its alternatives on multilayer perceptrons
John Waldo
Neural Network Deep Neural Network Comparative Study Multi Layer Back Propagation Previously Generated Alternative Differentiable Activation Function

May 13, 2022

Uninorm-like parametric activation functions for human-understandable neural models
Orsolya Csiszár, Luca Sára Pusztaházi, Lehel Dénes-Fazakas, Michael S. Gashler, Vladik Kreinovich, Gábor Csiszár
Deep Learning Model New Connection Fuzzy Logic Differentiable Activation Function Stable Relationship

March 16, 2022

Memorizing Transformers
Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, Christian Szegedy
Language Model Memory Trace Internal Representation Inference Time Differentiable Activation Function

Differentiable Activation Function

Papers

Hadamard Representations: Augmenting Hyperbolic Tangents in RL

Provable Bounds on the Hessian of Neural Networks: Derivative-Preserving Reachability Analysis

A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on $n$-dimensional Classes

Random Projection Neural Networks of Best Approximation: Convergence theory and practical applications

Leveraging Continuously Differentiable Activation Functions for Learning in Quantized Noisy Environments

STCSNN: High energy efficiency spike-train level spiking neural networks with spatio-temporal conversion

$\partial\mathbb{B}$ nets: learning discrete functions by gradient descent

Interpreting Deep Forest through Feature Contribution and MDI Feature Importance

A Robust Backpropagation-Free Framework for Images

A comparative study of back propagation and its alternatives on multilayer perceptrons

Uninorm-like parametric activation functions for human-understandable neural models

Memorizing Transformers