Back Propagation

Backpropagation is a fundamental algorithm for training artificial neural networks, primarily used to calculate gradients for updating network weights to minimize error. Current research focuses on improving backpropagation's efficiency and biological plausibility, exploring alternatives like forward-forward algorithms and methods that avoid the need for storing activations or full gradient calculations, often within the context of specific architectures such as transformers, spiking neural networks, and physics-informed neural networks. These efforts aim to reduce computational costs, memory requirements, and energy consumption, ultimately impacting the scalability and applicability of deep learning across various domains, including resource-constrained devices and large-scale models.

Papers

July 21, 2022

A Theoretical Framework for Inference and Learning in Predictive Coding Networks
Beren Millidge, Yuhang Song, Tommaso Salvatori, Thomas Lukasiewicz, Rafal Bogacz
LeArning Abstract Scientific Inference Back Propagation Neural Activity Predictive Coding Computational Neuroscience Theoretical Framework

July 19, 2022

Approximation Power of Deep Neural Networks: an explanatory mathematical survey
Mohammad Motamed
Deep Neural Network Deep Network Back Propagation Deep ReLU Network Nonlinear Function Approximation Approximation Power

July 15, 2022

An Exact Bitwise Reversible Integrator
Jos Stam
Back Propagation Computational Approach Differentiable Simulation Novel Single Integrator

July 12, 2022

Continual Learning with Deep Learning Methods in an Application-Oriented Context
Benedikt Pfülb
Deep Learning Artificial Intelligence Deep Neural Network Continual LEArning Back Propagation

July 8, 2022

Abs-CAM: A Gradient Optimization Interpretable Approach for Explanation of Convolutional Neural Networks
Chunyan Zeng, Kang Yan, Zhifeng Wang, Yan Yu, Shiyan Xia, Nan Zhao
Convolutional Neural Network Line by Line Explanation Back Propagation Saliency Map Gradient Based Fm G Cam

July 7, 2022

Backpropagation on Dynamical Networks
Eugene Tan, Débora Corrêa, Thomas Stemler, Michael Small
Back Propagation Network Inference Node Dynamic Dynamical Network

July 4, 2022

Automating the Design and Development of Gradient Descent Trained Expert System Networks
Jeremy Straub
Product Design Back Propagation Development Activity Expert System Neural Network System Rule Fact

July 2, 2022

A Structured Sparse Neural Network and Its Matrix Calculations Algorithm
Seyyed Mostafa Mousavi Janbeh Sarayi, Mansour Nikkhah Bahrami
Back Propagation Sparse Neural Network Sparse Matrix Large Matrix Matrix Computation

June 19, 2022

Characterizing and Mitigating the Difficulty in Training Physics-informed Artificial Neural Networks under Pointwise Constraints
Shamsulhaq Basir, Inanc Senocak
Neural Network Physic Informed Neural Network Back Propagation Difficulty Level Gradient Backpropagation Differential Operator Order Differential Operator

June 17, 2022

Learning the parameters of a differential equation from its trajectory via the adjoint equation
Imre Fekete, András Molnár, Péter L. Simon
Loss Function Gradient Descent Back Propagation Natural Gradient Many Parameter Differential Equation Temporal Trajectory Adjoint Method

June 6, 2022

Stacked unsupervised learning with a network architecture found by supervised meta-learning
Kyle Luther, H. Sebastian Seung
Unsupervised Learning Back Propagation Convolutional Kernel Unsupervised Algorithm Network Architecture Nondeterministic Stack

June 4, 2022

Classification at the Accuracy Limit -- Facing the Problem of Data Ambiguity
Claus Metzner, Achim Schilling, Maximilian Traxdorf, Konstantin Tziridis, Holger Schulze, Patrick Krauss
Classification Code Back Propagation Right Problem Classification Problem Classification Accuracy Accuracy Constraint Ambiguous Data

June 3, 2022

A Robust Backpropagation-Free Framework for Images
Timothy Zee, Alexander G. Ororbia, Ankur Mali, Ifeoma Nwogu
Back Propagation Deep Learning Algorithm Backward Propagation Differentiable Activation Function System Neuroscience

June 1, 2022

May 31, 2022

May 30, 2022

May 22, 2022

aSTDP: A More Biologically Plausible Learning
Shiyuan Li
Back Propagation Biological Neural Network Plausible Learning Biological Learning Dependent Plasticity

Back Propagation

Papers

A Theoretical Framework for Inference and Learning in Predictive Coding Networks

Approximation Power of Deep Neural Networks: an explanatory mathematical survey

An Exact Bitwise Reversible Integrator

Continual Learning with Deep Learning Methods in an Application-Oriented Context

Abs-CAM: A Gradient Optimization Interpretable Approach for Explanation of Convolutional Neural Networks

Backpropagation on Dynamical Networks

Automating the Design and Development of Gradient Descent Trained Expert System Networks

A Structured Sparse Neural Network and Its Matrix Calculations Algorithm

Characterizing and Mitigating the Difficulty in Training Physics-informed Artificial Neural Networks under Pointwise Constraints

Learning the parameters of a differential equation from its trajectory via the adjoint equation

Stacked unsupervised learning with a network architecture found by supervised meta-learning

Classification at the Accuracy Limit -- Facing the Problem of Data Ambiguity

A Robust Backpropagation-Free Framework for Images

B2T Connection: Serving Stability and Performance in Deep Transformers

A Theoretical Framework for Inference Learning

Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learning

A comparative study of back propagation and its alternatives on multilayer perceptrons

Backpropagation through Combinatorial Algorithms: Identity with Projection Works

Batch Normalization Is Blind to the First and Second Derivatives of the Loss

aSTDP: A More Biologically Plausible Learning