Surrogate Gradient

Surrogate gradient methods address the challenge of training neural networks with non-differentiable components, such as the threshold functions in spiking neural networks (SNNs). Current research focuses on improving the efficiency and accuracy of surrogate gradient algorithms, exploring variations like masked surrogate gradients and forward gradient injection, and investigating their theoretical underpinnings within both deterministic and stochastic SNNs. These advancements are significant for enabling efficient training of biologically-inspired SNNs for applications like speech recognition and object detection, as well as broader applications in areas like federated learning where only zeroth-order information is available. Ultimately, refined surrogate gradient techniques aim to bridge the gap between the energy efficiency of SNNs and the performance of traditional artificial neural networks.

Papers

June 28, 2024

Directly Training Temporal Spiking Neural Network with Sparse Surrogate Gradient
Yang Li, Feifei Zhao, Dongcheng Zhao, Yi Zeng
Neuromorphic Hardware Surrogate Gradient Sparse Gradient

May 31, 2024

Flexible and Efficient Surrogate Gradient Modeling with Forward Gradient Injection
Sebastian Otte
Deep Learning Framework Extension Study Automatic Differentiation Surrogate Gradient Forward Gradient Gradient Manipulation Gradient Amplification

April 23, 2024

Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks
Julia Gygax, Friedemann Zenke
Neural Network Natural Gradient Neuromorphic Computing Automatic Differentiation Single Neuron Surrogate Gradient Theoretical Foundation

April 22, 2024

Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks
Alexandre Bittar, Philip N. Garner
Neural Network Neural Activity Neural Dynamic Speech Perception Neural Oscillation Surrogate Gradient Neuromorphic Technology

January 14, 2024

Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy
Chengli Tan, Jiangshe Zhang, Junmin Liu, Yicheng Wang, Yunda Hao
Gradient Descent Sharpness Aware Minimization Renormalization Group Surrogate Gradient Exact Gradient Sharpness Aware Training

October 18, 2023

Optimising Distributions with Natural Gradient Surrogates
Jonathan So, Richard E. Turner
Natural Gradient Surrogate Modeling Surrogate Gradient

September 7, 2023

Enabling energy-Efficient object detection with surrogate gradient descent in spiking neural networks
Jilong Luo, Shanlin Xiao, Yinsheng Chen, Zhiyi Yu
Neural Network Object Detection Spiking Neural Network Look Once, Version 3 Surrogate Gradient

August 8, 2023

Federated Zeroth-Order Optimization using Trajectory-Informed Surrogate Gradients
Yao Shu, Xiaoqiang Lin, Zhongxiang Dai, Bryan Kian Hsiang Low
Federated Prompt Cooperation Zeroth Order Federated Optimization Surrogate Gradient

April 26, 2023

Membrane Potential Distribution Adjustment and Parametric Surrogate Gradient in Spiking Neural Networks
Siqi Wang, Tee Hiang Cheng, Meng-Hiot Lim
Spiking Neural Network Surrogate Gradient Membrane Potential

April 25, 2023

Uncovering the Representation of Spiking Neural Networks Trained with Surrogate Gradient
Yuhang Li, Youngeun Kim, Hyoungseob Park, Priyadarshini Panda
Spiking Neural Network Meaningful Representation Individual Representation Neural Network Representation Surrogate Gradient

February 18, 2023

KLIF: An optimized spiking neuron unit for tuning surrogate gradient slope and membrane potential
Chunming Jiang, Yilei Zhang
Neuron Model Processing Unit Surrogate Gradient Leaky Integrate and Fire Neuromorphic Network Membrane Potential Spike Based Backpropagation

February 24, 2022

Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting
Shikuang Deng, Yuhang Li, Shanghang Zhang, Shi Gu
Spiking Neural Network Dynamic Reweighting Neuronal Network Surrogate Gradient Backward Training

February 1, 2022

Stabilizing Spiking Neuron Training
Luca Herranz-Celotti, Jean Rouat
Neuromorphic Computing Surrogate Gradient

January 26, 2022

Meta-learning Spiking Neural Networks with Surrogate Gradient Descent
Kenneth Stewart, Emre Neftci
Spiking Neural Network Neuromorphic Hardware Gradient Based Meta Learning SNN Based Surrogate Gradient Neuromorphic Learning

January 3, 2022

Learning with Latent Structures in Natural Language Processing: A Survey
Zhaofeng Wu
LeArning Abstract Natural Language Processing Timely Survey Inductive Bias Latent Structure Differentiable Model Surrogate Gradient Discrete Latent

Surrogate Gradient

Papers

Directly Training Temporal Spiking Neural Network with Sparse Surrogate Gradient

Flexible and Efficient Surrogate Gradient Modeling with Forward Gradient Injection

Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks

Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks

Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy

Optimising Distributions with Natural Gradient Surrogates

Enabling energy-Efficient object detection with surrogate gradient descent in spiking neural networks

Federated Zeroth-Order Optimization using Trajectory-Informed Surrogate Gradients

Membrane Potential Distribution Adjustment and Parametric Surrogate Gradient in Spiking Neural Networks

Uncovering the Representation of Spiking Neural Networks Trained with Surrogate Gradient

KLIF: An optimized spiking neuron unit for tuning surrogate gradient slope and membrane potential

Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

Stabilizing Spiking Neuron Training

Meta-learning Spiking Neural Networks with Surrogate Gradient Descent

Learning with Latent Structures in Natural Language Processing: A Survey