Backpropagation

Follow this topic Ask about this topic

Backpropagation uses the chain rule to send error signals from a model’s outputs backward through its layers, providing gradients for updating parameters. Work improves its efficiency and reach through memory-saving gradient methods, parallel scans, gradient approximations, and better scheduling, including for recurrent models and large language models. Alternatives such as feedback alignment, predictive coding, and forward-only local learning aim to reduce global backward communication or support biological and physical hardware. The central tension is that these alternatives may improve locality or memory use, but often struggle to match backpropagation’s accuracy, convergence, or measured end-to-end energy efficiency.

Latest papers

44 papers5 papers in the last four weeks, down 17% on the four weeks before. 0.0% of all new papers.

Neural LoFi with Backward Coupling: A Spectral Theory of Cross-Layer Feature Learning
Oct 23, 2026Yatin Dandi, Matteo Vilucchio, Hugo Tabanelli +1Neural Network Training Dynamics Backpropagation
Information, Learning, and Physics Laboratory (IDEPHICS) EPFL, Lausanne, Switzerland
How Much Gradient Does Truncated Backprop Lose in Looped Transformers?
Oct 20, 2026Aur Shalev MerinTruncated Backpropagation Through Time Backpropagation
Independent researcher
FidLIF: Parallel Training of High-Fidelity Spiking Neurons with Exact BPTT
Oct 12, 2026Kaixiang He, Yujie WuLIF Neurons Spiking Neural Networks
Department of Computing, The Hong Kong Polytechnic University
Connected Self Forcing: Beyond Local Learning in Video Autoregression
Oct 8, 2026Dongbin Zhang, Chaoda Zheng, Kangjie Chen +13Autoregressive Video Generation Streaming Video Generation
XPeng · The Chinese University of Hong Kong · Tsinghua University
Backward-State Policy Is Part of the Learning Algorithm
Sep 30, 2026Shuxiao Xie, Shuyang Xie, Dezhi Ran +2Mixed-Precision Quantization Backpropagation
Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC), China
The Price of Locality: Why Forward-Forward Underperforms Backpropagation?
Sep 27, 2026Zhaoxian Wu, Haichuan Liu, Tianyi ChenForward-Forward Learning Neural Network Optimization
Cornell Tech and Cornell University
LocalProp: Neuro-Localized Memory-Efficient Backpropagation
Sep 26, 2026Diana-Nicoleta Grigore, Iuliana Georgescu, Radu Tudor IonescuLocal Learning Backpropagation
Department of Computer Science University of Bucharest, Romania
The Ups and Downs of Backprop Weights
Sep 18, 2026Giuseppe Chindemi, Benjamin F. GreweModular Neural Networks Dynamic Neural Networks
Institute of Neuroinformatics, UZH / ETH Zurich ETH AI Center Zurich, Switzerland
Large Distant Gradients Need Not Be Reliable: reliability-weighted credit assignment for long-horizon autoregressive forecasting
Sep 11, 2026Junhao Zhao, David Michael Simberg, Jacob Kang +2Long-Term Time Series Forecasting Backpropagation
University of Maryland, College Park
Learning Length-Extrapolatable Recurrent Models
Sep 8, 2026Hanwen JiangBackpropagation Temporal Credit Assignment
Adobe Research
Wave Function Backpropagation with Explicit Temporal-Interval Dynamics
Sep 1, 2026Byunggu Yu, Justin KimBackpropagation Feedforward Neural Networks
Dept. of Computer Science and Information Technology University of the District of Columbia Washington, DC USA
The Forward-Backward Disconnect: State Dynamics, Credit Assignment, and Biological Grounding in Neural Computation
Aug 20, 2026Hadi Al Mubasher, Mariette AwadCredit Assignment Neural Network Training Dynamics
Department of Electrical and Computer Engineering, Maroun Semaan Faculty of Engineering and Architecture, American University of Beirut, Lebanon
ERASE: EaRly bAckpropagation SchEdule for Faster Training of Modern Recommendation Systems
Aug 19, 2026Ergan Shang, Flavio Sales TruzziDeep Learning Optimization Backpropagation
Carnegie Mellon University · Meta Inc.
Unifying Physical Backpropagation
Aug 12, 2026Cyrill Bösch, Yigithan Gediz, Clara C. Wanjura +1On-Device Training Backpropagation
Department of Computer Science, Princeton University, Princeton, NJ 08540, USA · Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08540, USA
Learning in Deep Networks under Dale's Constraint
Aug 7, 2026Roy Abel, Shimon UllmanBackpropagation Biologically Plausible Learning
Weizmann Institute of Science
Reconstructing Backpropagation from Forward Fluctuations in Noise-modulated Neural Networks
Jul 29, 2026Shuhei IkemotoBackpropagation Neuromorphic Computing
Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, Kitakyushu, Japan
Continual-RL for Generalization in Autonomous Racing on the RoboRacer Platform
Jul 27, 2026Joel Siegert, Edoardo Ghignone, Michele MagnoBackpropagation Continual Robot Learning
Center for Project-Based Learning, D-ITET, ETH Zurich
JAGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Models
Jul 20, 2026Ruiyi Ding, Jie Li, He Kang +3Backpropagation Diffusion Transformer
1KlingAI Research · University of Science and Technology of China · 2Fudan University +2
Building a Neural Network from Scratch: Implementation, Evaluation, and Optimization
Jul 18, 2026Yuanzhe JiaMultilayer Perceptrons Backpropagation
University of Sydney, Australia
Backpropagation-Free Trunk Training via the Split Forward Gradients
Jul 18, 2026Tian Qin, Wei-Min HuangBackpropagation Gradient Descent
Math department Lehigh University Bethlehem, PA 18015, USA
Backpropagation as a Nilpotent Linear System
Jul 13, 2026Ahmed BoughammouraBackpropagation Neural Network Training Dynamics
Higher Institute of Informatics and Mathematics of Monastir, University of Monastir, 5000 Monastir, Tunisia.
K-ABENA: K-Adaptive Backpropagation with Error-based N-exclusion Algorithm : (Compensated Loss-Based Sample Exclusion with Unbiased Gradient Estimation)
Jul 7, 2026Jean-Francois BonbhelBackpropagation Neural Network Optimization
NeuroSoft IA, Qu´ebec City, Canada · YekoElite University, Brazzaville, Republic of Congo
FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training
Jun 22, 2026Dikshant Kukreja, Kritarth Prasad, Avinash Anand +6Automatic Differentiation Backpropagation
Puch AI · Singapore Institute of Technology · Nanyang Technological University +2
Automatic Differentiation from Scratch: How PyTorch Computes Gradients in Physics-Informed Neural Networks
Jun 13, 2026Abdeladhim TahimiAutomatic Differentiation Backpropagation
CECA, Universidade Federal de Alagoas (UFAL), Rio Largo, Alagoas, 57100-000, Brazil
Overcoming Rank Collapse in Feedback Alignment
Jun 9, 2026Gauthier Boeshertz, Razvan Pascanu, Claudia ClopathBackpropagation Neural Network Optimization
Department of Bioengineering Imperial College London · Mila
Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics
Jun 8, 2026Claudio NordioBackpropagation ReLU Neural Networks
Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training
Jun 4, 2026Yucheng ChenBackpropagation Local Learning
Amplimit Toronto, ON, Canada
Augmented Lagrangian Predictive Coding
May 29, 2026Jeffrey Seely, Julian GouldBackpropagation Predictive Coding
Sakana AI
PINNs Failure Modes are Overfitting
May 29, 2026Nigel T. Andersen, Takashi MatsubaraNeural Network Generalization Neural PDE Solvers
Graduate School of Information Science and Technology Hokkaido University Sapporo, Japan · RIKEN Center for Advanced Intelligence Project (AIP) Tokyo, Japan
Misalignment Between Backpropagation and the Hierarchy of Brain Responses to Images
May 27, 2026Joséphine Raugel, Maximilian Seitzer, Marc Szafraniec +6Backpropagation Visual Representation Learning
Meta AI · Ecole Normale Supérieure, PSL University

Oct 23, 2026J/K move · Enter open · S save

Neural LoFi with Backward Coupling: A Spectral Theory of Cross-Layer Feature Learning

Yatin Dandi, Matteo Vilucchio, Hugo Tabanelli, Florent Krzakala

Information, Learning, and Physics Laboratory (IDEPHICS) EPFL, Lausanne, Switzerland

Neural Network Training Dynamics Backpropagation

Open paper AskPDF arXiv

During joint training, backpropagation allows later layers to reshape earlier representations, a feedback mechanism absent from purely layer-wise learning. We extend Neural Low-Degree Filtering (Neural LoFi), a recently proposed layer-wise spectral surrogate for feature learning, to capture this interaction. Starting from the exact backpropagation gradients, we derive a local coupling between a neuron's incoming weights and its corresponding next-layer column. In a controlled small-initialization regime with smooth activations and fixed surroundings, these dynamics amplify singular modes of a cross-layer correlation operator. Each growing mode pairs an earlier feature with an upper-layer response through shared neuron coefficients. This motivates a backward-coupling algorithm in which upper-layer sensitivities generate sample-dependent effective labels for earlier spectral steps, while corrected features refine upward transmission. A function-space variational principle characterizes the selected feature-response pairs by maximizing the label correlation of their product under kernel-norm constraints. For three-layer networks, we derive self-consistent continuous-time dynamics for the filtered hidden-layer features, the activation kernels, and the network output under the proposed backward coupling. In a hierarchical multi-index model, we establish a feature-recovery separation: backward coupling recovers lower-layer directions invisible to the original forward label-weighted statistic. The linear specialization recovers residual-dependent singular-mode dynamics, while paired modes provide a dynamical mechanism for linked low-rank covariance components consistent with the rainbow-network picture. Experiments with fully connected and convolutional networks on binary CIFAR-10 show improved accuracy over forward Neural LoFi at matched widths.