Backward Propagation

Backward propagation (BP), a cornerstone of training artificial neural networks, aims to efficiently compute gradients for updating model parameters. Current research focuses on improving BP's efficiency and applicability across diverse architectures, including transformers, spiking neural networks, and optical neural networks, exploring alternatives like forward-only methods and modifications to reduce computational cost and memory requirements. These advancements are crucial for training increasingly complex models, enabling progress in areas like large language models, neuromorphic computing, and efficient solutions to partial differential equations.

Papers

December 2, 2022

Credit Assignment for Trained Neural Networks Based on Koopman Operator Theory
Zhen Liang, Changyuan Zhao, Wanwei Liu, Bai Xue, Wenjing Yang, Zhengbin Pang
Neural Network Koopman Operator Neural Network Architecture Backward Propagation Credit Assignment

November 9, 2022

Accelerating Adversarial Perturbation by 50% with Semi-backward Propagation
Zhiqi Bu
Adversarial Robustness Adversarial Perturbation Backward Propagation Gradient Based Parameter

October 18, 2022

Exact Gradient Computation for Spiking Neural Networks Through Forward Propagation
Jane H. Lee, Saeid Haghighatshoar, Amin Karbasi
Spike Train Backward Propagation Exact Gradient

September 30, 2022

An In-depth Study of Stochastic Backpropagation
Jun Fang, Mingze Xu, Hao Chen, Bing Shuai, Zhuowen Tu, Joseph Tighe
Deep Neural Network Stochastic Gradient Comprehensive Study Backward Propagation Stochastic Neural Network Dropout Regularization

September 20, 2022

Towards Fine-tuning Pre-trained Language Models with Integer Forward and Backward Propagation
Mohammadreza Tayaranian, Alireza Ghaffari, Marzieh S. Tahaei, Mehdi Rezagholizadeh, Masoud Asgharian, Vahid Partovi Nia
Language Model Fine Tuning Pre Trained Language Model Backward Propagation Fine Tuned BERT Integer Only Training

September 19, 2022

State-driven Implicit Modeling for Sparsity and Robustness in Neural Networks
Alicia Y. Tsai, Juliette Decugis, Laurent El Ghaoui, Alper Atamtürk
Neural Network Native Robustness Sparsity Increase Learning Model Implicit Model Backward Propagation Implicit Differentiation

August 29, 2022

ANAct: Adaptive Normalization for Activation Functions
Yuan Peiwen, Henan Liu, Zhu Changsheng, Yuyi Wang
Adaptive Normalization Backward Propagation Gradient Consistency

August 5, 2022

DIVISION: Memory Efficient Training via Dual Activation Precision
Guanchu Wang, Zirui Liu, Zhimeng Jiang, Ninghao Liu, Na Zou, Xia Hu
Deep Neural Network Backward Propagation DNN Training High Compression Memory Efficient Training

July 10, 2022

A Forward Propagation Algorithm for Online Optimization of Nonlinear Stochastic Differential Equations
Ziheng Wang, Justin Sirignano
Stochastic Differential Equation Online Optimization Backward Propagation

July 7, 2022

Style Interleaved Learning for Generalizable Person Re-identification
Wentao Tan, Changxing Ding, Pengfei Wang, Mingming Gong, Kui Jia
Person Re Identification Backward Propagation Interleaving Method Generalizable Person Re Identification Interleaved Learning

June 3, 2022

A Robust Backpropagation-Free Framework for Images
Timothy Zee, Alexander G. Ororbia, Ankur Mali, Ifeoma Nwogu
Back Propagation Deep Learning Algorithm Backward Propagation Differentiable Activation Function System Neuroscience

April 4, 2022

Unsupervised Learning of Accurate Siamese Tracking
Qiuhong Shen, Lei Qiao, Jinyang Guo, Peixia Li, Xin Li, Bo Li, Weitao Feng, Weihao Gan, Wei Wu, Wanli Ouyang
Unsupervised Learning Unsupervised Method Visual Object Tracking Backward Propagation Siamese Tracker

March 26, 2022

SlimFL: Federated Learning with Superposition Coding over Slimmable Neural Networks
Won Joon Yun, Yunseok Kwak, Hankyul Baek, Soyi Jung, Mingyue Ji, Mehdi Bennis, Jihong Park, Joongheon Kim
Federated Learning Backward Propagation Slimmable Neural Network

March 18, 2022

Randomized Sharpness-Aware Training for Boosting Computational Efficiency in Deep Learning
Yang Zhao, Hao Zhang, Xiuyuan Hu
Deep Learning Computational Efficiency Backward Propagation Sharpness Aware Training

January 15, 2022

Physical Derivatives: Computing policy gradients by physical forward-propagation
Arash Mehrjou, Ashkan Soleymani, Stefan Bauer, Bernhard Schölkopf
Policy Gradient Model Based Reinforcement Learning Model Free Dynamic Model Physical System Backward Propagation Derivative Process Custom Robot

December 23, 2021

Forward Composition Propagation for Explainable Neural Reasoning
Isel Grau, Gonzalo Nápoles, Marilyn Bello, Yamisleydi Salgueiro, Agnieszka Jastrzebska
Neural Network Pattern Recognition Backward Propagation

December 20, 2021

Accurate online training of dynamical spiking neural networks through Forward Propagation Through Time
Bojian Yin, Federico Corradi, Sander M. Bohte
Neural Network Time Matter Recurrent Network Neuron Model Backward Propagation Standard Deep Online Training Liquid Time Constant Backpropagation Through Time

December 5, 2021

Joint Superposition Coding and Training for Federated Learning over Multi-Width Neural Networks
Hankyul Baek, Won Joon Yun, Yunseok Kwak, Soyi Jung, Mingyue Ji, Mehdi Bennis, Jihong Park, Joongheon Kim
Training Data Federated Learning Backward Propagation Width Neural Network Slimmable Neural Network

November 22, 2021

Branching Time Active Inference: the theory and its generality
Théophile Champion, Lancelot Da Costa, Howard Bowman, Marek Grześ
Theoretical Understanding Monte Carlo Tree Search Active Inference Exact Inference Backward Propagation Habitual Behavior

November 10, 2021

Efficient Neural Network Training via Forward and Backward Propagation Sparsification
Xiao Zhou, Weizhong Zhang, Zonghao Chen, Shizhe Diao, Tong Zhang
Neural Network Sparse Training Sparse Structure Backward Propagation Sparsity Constraint