Backpropagation
Overview
Backpropagation uses the chain rule to send error signals from a model’s outputs backward through its layers, providing gradients for updating parameters. Work improves its efficiency and reach through memory-saving gradient methods, parallel scans, gradient approximations, and better scheduling, including for recurrent models and large language models. Alternatives such as feedback alignment, predictive coding, and forward-only local learning aim to reduce global backward communication or support biological and physical hardware. The central tension is that these alternatives may improve locality or memory use, but often struggle to match backpropagation’s accuracy, convergence, or measured end-to-end energy efficiency.
Latest papers
48 papers6 papers in the last four weeks, up 20% on the four weeks before. 0.0% of all new papers.
Augmented Lagrangian Predictive Coding
PINNs Failure Modes are Overfitting
Misalignment Between Backpropagation and the Hierarchy of Brain Responses to Images
Is Backpropagation Optimal? When Synthetic Gradients Improve Sample Efficiency
Understanding Sample Efficiency in Predictive Coding
Efficient and provably convergent end-to-end training of deep neural networks with linear constraints
LBI: Parallel Scan Backpropagation via Latent Bounded Interfaces
Biological Plausibility and Representational Alignment of Feedback Alignment in Convolutional Networks
Physics-Informed Neural Networks: A Didactic Derivation of the Complete Training Cycle
Untrained CNNs Match Backpropagation at V1: A Systematic RSA Comparison of Four Learning Rules Against Human fMRI
Local learning for stable backpropagation-free neural network training towards physical learning
A Variational Latent Equilibrium for Learning in Neuronal Circuits
BASIS: Balanced Activation Sketching with Invariant Scalars for "Ghost Backpropagation"
Riemannian Optimization in Modular Systems
Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation
Energy-Efficient Deep Learning Without Backpropagation: A Rigorous Hardware-Validated Benchmarking Study of Forward-Only Algorithms
Perforated Backpropagation: A Neuroscience Inspired Extension to Artificial Neural Networks
perforated'' backpropagation empowering the artificial neurons of deep neural networks to achieve better performance coding for the same features they coded for in the original architecture. After an initial network training phase, additional dendrite'' nodes are added to the network and separately trained with a different objective: to correlate their output with the remaining error of the original neurons. The trained dendrites are then frozen, and the original neurons are further trained, now taking into account the additional error signals provided by the dendrites. The cycle of training the original neurons and then adding and training dendrites can be repeated several times until satisfactory performance is achieved. Our algorithm was successfully added to modern state-of-the-art PyTorch networks across multiple domains, improving upon original accuracies and allowing for significant model compression without a loss in accuracy.