Orthogonal Weight Modification Enhances Learning Scalability and Convergence Efficiency without Gradient Backpropagation

Guoqing Ma; Shan Yu

cs.LGFeb 25, 2026

Orthogonal Weight Modification Enhances Learning Scalability and Convergence Efficiency without Gradient Backpropagation

Authors: Guoqing Ma, Shan Yu

Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Future Technology, University of Chinese Academy of Sciences · Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Chinese Academy of Sciences

Abstract

Recognizing the substantial computational cost of backpropagation (BP), non-BP methods have emerged as attractive alternatives for efficient learning on emerging neuromorphic systems. However, existing non-BP approaches still face critical challenges in efficiency and scalability. Inspired by neural representations and dynamic mechanisms in the brain, we propose a perturbation-based approach called LOw-rank Cluster Orthogonal (LOCO) weight modification. We find that low-rank is an inherent property of perturbation-based algorithms. Under this condition, the orthogonality constraint limits the variance of the node perturbation (NP) gradient estimates and enhances the convergence efficiency. Through extensive evaluations on multiple datasets, LOCO demonstrates the capability to locally train the deepest spiking neural networks to date (more than 10 layers), while exhibiting strong continual learning ability, improved convergence efficiency, and better task performance compared to other brain-inspired non-BP algorithms. Notably, LOCO requires only O(1) parallel time complexity for weight updates, which is significantly lower than that of BP methods. This offers a promising direction for achieving high-performance, real-time, and lifelong learning on neuromorphic systems.

Figures & tables

Figure 1: Schematic diagram of LOCO. (A) The multi-layer architecture of SNN. The weights connecting to the same postsynaptic neurons are marked with the same color, representing the unit of weight modification in LOCO (Eq. 4 ). (B) The weight modification with NP and LOCO. (C) During the training of new tasks, the NP weight modification $\Delta{{\rm{W}}^{NP}}$ , is projected to the subspace (green surface), in which good performance for old tasks can be maintained. As a result, the actually implemented weight modification is $\Delta{{\rm{W}}^{LOCO}}$ . (D) The process of calculating the projection matrix $P_{l}$ . (E) The reduced variation during the parameter searching process between NP and LOCO. NP searches in a high-dimensional space (orange line) with high variance. LOCO reduces the variance and searches in a low-dimensional subspace (green planes).

Figure 2: LOCO demonstrates superior scalability, enhanced convergence efficiency, and continual learning capabilities without gradient backpropagation. (A to C) Performance on the MNIST dataset. (A) LOCO can train a deeper spiking neural network than STDP+SBP and NP. Performance of network with differrent hidden layers are color-coded. (B) Learning dynamics of LOCO for neural networks ranging from 3 to 10 layers. The inset shows the accuracy curve of NP. (C) Accuracy curves for continual learning task on MNIST. The horizontal axis represents the current class being learned, with each class trained sequentially. The vertical axis indicates the average classification accuracy on all previously learned classes. (D to F) Performance in the phonetic transcription task, presented in the same manner as that in (A) and (B). The shaded regions and error bars represent the variance of the performance across five runs with different seeds.

Figure 3: Low-rank is an inherent property of perturbation-based algorithm.

Figure 4: The magnitude of weight changes in LOCO is smaller than that of NP. This implies that LOCO is more stable and energy-efficient.

Neuromorphic Computing

Neuromorphic computing uses brain-inspired, event-driven algorithms and hardware to process information with low power and latency.

See all 194 papers

Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses
May 20, 2025Nazmus Saadat As-Saquib, A N M Nafiz Abeer, Hung-Ta Chien +3Neural Network Optimization Local Learning
Texas A&M University, College Station, TX 77843 · Sandia National Laboratories, Livermore, CA 94550
LocalProp: Neuro-Localized Memory-Efficient Backpropagation
Sep 26, 2026Diana-Nicoleta Grigore, Iuliana Georgescu, Radu Tudor IonescuLocal Learning Backpropagation
Department of Computer Science University of Bucharest, Romania
Breaking chains with trees: Deep learning with $\mathcal{O}(\log N)$ parallel time complexity
Jun 19, 2026Neeraj Mohan Sushma, Aditya Nagarsekar, Cabrel Teguemne Fokam +4Deep Learning Optimization Neural Network Optimization
Center for Cognitive Interaction Technology, Bielefeld University, Germany · Ruhr Universität Bochum, Germany · Birla Institute of Technology and Science, Pilani, Goa Campus +1

May 20, 2025J/K move · Enter open · S save

Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses

Nazmus Saadat As-Saquib, A N M Nafiz Abeer, Hung-Ta Chien, Byung-Jun Yoon+2

Texas A&M University, College Station, TX 77843 · Sandia National Laboratories, Livermore, CA 94550

Neural Network Optimization Local Learning

Open paper AskPDF arXiv

Training neural networks has traditionally relied on backpropagation (BP), a gradient-based algorithm that, despite its widespread success, suffers from key limitations in both biological and hardware perspectives. These include backward error propagation by symmetric weights, non-local credit assignment, and frozen activity during backward passes. We propose Forward Target Propagation (FTP), a biologically plausible and computationally efficient alternative that replaces the backward pass with a second forward pass. FTP estimates layerwise targets using only feedforward computations, eliminating the need for symmetric feedback weights or learnable inverse functions, hence enabling modular and local learning. We evaluate FTP on fully connected networks, CNNs, and RNNs, demonstrating accuracies competitive with BP on MNIST, CIFAR10, and CIFAR100, as well as effective modeling of long-term dependencies in sequential tasks. Moreover, FTP outperforms BP under quantized low-precision and emerging hardware constraints while also demonstrating substantial efficiency gains over other biologically inspired methods such as target propagation variants and forward-only learning algorithms. With its minimal computational overhead, forward-only nature, and hardware compatibility, FTP provides a promising direction for energy-efficient on-device learning and neuromorphic computing.

Abstract

Neuromorphic Computing

Explore similar work

Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses

LocalProp: Neuro-Localized Memory-Efficient Backpropagation

Breaking chains with trees: Deep learning with O(log⁡N)\mathcal{O}(\log N)O(logN) parallel time complexity

Breaking chains with trees: Deep learning with $\mathcal{O}(\log N)$ parallel time complexity