Local learning for stable backpropagation-free neural network training towards physical learning

Yaqi Guo; Fabian Braun; Bastiaan Ketelaar; Stephanie Tan; Richard Norte; Siddhant Kumar

cs.LGMar 25, 2026

Local learning for stable backpropagation-free neural network training towards physical learning

Authors: Yaqi Guo, Fabian Braun, Bastiaan Ketelaar, Stephanie Tan, Richard Norte, Siddhant Kumar

Organizations: Department of Materials Science and Engineering, Delft University of Technology, Mekelweg 2, Delft, 2628CD, The Netherlands · Department of Precision and Microsystems Engineering, Delft University of Technology, Mekelweg 2, Delft, 2628CD, The Netherlands · Department of Intelligent Systems, Delft University of Technology, Van Mourik Broekmanweg 6, Delft 2628XE, The Netherlands

Abstract

While backpropagation and automatic differentiation have driven deep learning's success, the physical limits of chip manufacturing and rising environmental costs of deep learning motivate alternative learning paradigms such as physical neural networks. However, most existing physical neural networks still rely on digital computing for training, largely because backpropagation and automatic differentiation are difficult to realize in physical systems. We introduce FFzero, a forward-only learning framework enabling stable neural network training without backpropagation or automatic differentiation. FFzero combines layer-wise local learning, prototype-based representations, and directional-derivative-based optimization through forward evaluations only. We show that local learning is effective under forward-only optimization, where backpropagation fails. FFzero generalizes to multilayer perceptron and convolutional neural networks across classification and regression. Using a simulated photonic neural network as an example, we demonstrate that FFzero provides a viable path toward backpropagation-free in-situ physical learning.

Figures & tables

Figure 1: Schematic comparison of FFzero and backpropagation performance when trained with directional derivatives. FFzero employs a local learning strategy in which each layer is updated independently, preventing gradient-estimation errors from compounding across layers. In contrast, backpropagation propagates directional-derivative estimation errors globally across layers, causing performance to degrade with increasing model size. FFzero maintains stable performance at scale while remaining compatible with in-situ physical implementation.

Figure 2: Schematic of FFzero algorithm, illustrated for a (a) 3-class classification problem and (b) regression problem. (a) Each linear layer is trained locally and independently by maximizing layer-wise goodness. The cosine similarities of the layer output w.r.t. the fixed prototype vectors (denoted by $\xi[\cdot]$ ) are used to compute the goodness. Directional derivatives are used to perform gradient-based optimization for each layer locally. (b) In regression tasks, the antipodal prototypes represent the upper ( $y=1$ ) and lower ( $y=-1$ ) prediction bounds, with intermediate ground-truth values encoded as interpolations between them. Each layer’s output is normalized onto a unit sphere and compared against the fixed antipodal vectors via cosine similarity, yielding layer-wise goodness for weights optimization via directional derivatives.

Figure 3: Schematic of directional derivative optimization in FFzero. (a) Illustration of directional derivative (DD) estimation on a representative goodness landscape. (b) Gradient estimation by averaging directional derivatives over multiple random directions. (c) Comparison of optimization trajectories obtained using DD-based updates and standard gradient ascent. While gradient ascent follows the true gradient direction, the expectation of the DD-based update matches the true gradient update (Supplementary Information 2). (d) Schematic illustration of the goodness-based DD update for a linear layer.

Figure 4: Classification accuracy of MLP models across varying hidden-layer widths, with all hidden layers constrained to equal size. Results are shown for (a) MNIST and (b) FashionMNIST datasets using backpropagation (BP) and forward–forward (FF) training paradigms, each combined with either automatic differentiation (AD) or directional-derivative (DD) optimization. The numbers above each plot indicate the total number of parameters for the corresponding model.

Figure 5: Classification accuracy of CNN models across varying number of channels per convolution layer. Results are shown for MNIST and FashionMNIST datasets using backpropagation (BP) and forward–forward (FF) training paradigms, each combined with either automatic differentiation (AD) or directional-derivative (DD) optimization. The numbers above each plot indicate the total number of parameters for the corresponding model.

Figure 6: Regression accuracy of MLP and CNN models. (a,b) Results are shown for regression on synthetic functions in equation ( 15 ) using MLPs across varying hidden-layer widths, with all hidden layers constrained to equal size. (c,d) Results are shown for regression on MNIST using MLP and CNN of varying hidden-layer widths and number of channels per convolution layer, respectively. For all cases, the comparison is made between backpropagation (BP) and forward–forward (FF) training paradigms, each combined with either automatic differentiation (AD) or directional-derivative (DD) optimization. The numbers above each plot indicate the total number of parameters for the corresponding model.

Figure 7: (a) Schematic of FFzero training applied to a two-layer (emulated) photonic neural network. (b) Test accuracy on MNIST using BP+AD, BP+DD, FF+AD, and FF+DD (FFzero).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Classification accuracy of MLP models across varying number of sampled directions used in directional-derivative (DD) optimization. Results are shown for MNIST dataset with MLPs of fixed depth $d=5$ and different hidden-layer widths: $w\in\{10,50,100\}$ , with all hidden layers constrained to equal size. The results are compared between backpropagation (BP) and forward–forward (FF) training paradigms, each combined with either automatic differentiation (AD) or directional-derivative (DD) optimization.

Figure 2: PCA projection of the layer-wise embeddings of a two-layer photonic neural network trained with FFzero on MNIST. Each point represents one test sample, colored by the digit class. Stars denote the pre-computed prototype vectors projected into the same PCA space. After training with FFzero, data points cluster towards their corresponding prototype vectors in all layers.

Figure 3: Schematic illustration of the FFzero algorithm and architecture used for MLPs in ( a ) classification and ( b ) regression tasks.

Figure 4: Schematic illustration of the FFzero algorithm and architecture used for CNNs in ( a ) classification and ( b ) regression tasks. Within each convolution layer, feature maps from individual channels are reshaped and projected to a lower-dimensional space via fixed random linear mappings. The reduced representations are compared with fixed prototype vectors using cosine similarity, and a channel-wise goodness is computed to update the corresponding channel parameters through DD.

Parameter	Notation	Value
MLP for MNIST classification, MNIST regression,
and FashionMNIST classification:
Size of training dataset	-	$50000$
Size of test dataset	-	$10000$
Batch size	-	$256$
Number of training epochs	-	$100$

Appendix

Table 1: List of parameters and hyperparameters used in experiments.

Parameter	Notation	Value
CNN for FashionMNIST classification:
Size of training dataset	-	$50000$
Size of test dataset	-	$10000$
Number of training epochs	-	$100$
Learning rate	$\lambda$	$0.001$
Step size in DD	$\epsilon$	$0.001$

Appendix

Table 1: (continued) List of parameters and hyperparameters used in experiments.

Physical Neural Networks

See all 10 papers

Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses
May 20, 2025Nazmus Saadat As-Saquib, A N M Nafiz Abeer, Hung-Ta Chien +3Neural Network Optimization Local Learning
Texas A&M University, College Station, TX 77843 · Sandia National Laboratories, Livermore, CA 94550
Reconstructing Backpropagation from Forward Fluctuations in Noise-modulated Neural Networks
Jul 29, 2026Shuhei IkemotoBackpropagation Neuromorphic Computing
Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, Kitakyushu, Japan
LocalProp: Neuro-Localized Memory-Efficient Backpropagation
Sep 26, 2026Diana-Nicoleta Grigore, Iuliana Georgescu, Radu Tudor IonescuLocal Learning Backpropagation
Department of Computer Science University of Bucharest, Romania

May 20, 2025J/K move · Enter open · S save

Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses

Nazmus Saadat As-Saquib, A N M Nafiz Abeer, Hung-Ta Chien, Byung-Jun Yoon+2

Texas A&M University, College Station, TX 77843 · Sandia National Laboratories, Livermore, CA 94550

Neural Network Optimization Local Learning

Open paper AskPDF arXiv

Training neural networks has traditionally relied on backpropagation (BP), a gradient-based algorithm that, despite its widespread success, suffers from key limitations in both biological and hardware perspectives. These include backward error propagation by symmetric weights, non-local credit assignment, and frozen activity during backward passes. We propose Forward Target Propagation (FTP), a biologically plausible and computationally efficient alternative that replaces the backward pass with a second forward pass. FTP estimates layerwise targets using only feedforward computations, eliminating the need for symmetric feedback weights or learnable inverse functions, hence enabling modular and local learning. We evaluate FTP on fully connected networks, CNNs, and RNNs, demonstrating accuracies competitive with BP on MNIST, CIFAR10, and CIFAR100, as well as effective modeling of long-term dependencies in sequential tasks. Moreover, FTP outperforms BP under quantized low-precision and emerging hardware constraints while also demonstrating substantial efficiency gains over other biologically inspired methods such as target propagation variants and forward-only learning algorithms. With its minimal computational overhead, forward-only nature, and hardware compatibility, FTP provides a promising direction for energy-efficient on-device learning and neuromorphic computing.

Abstract

Physical Neural Networks

Explore similar work

Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses

Reconstructing Backpropagation from Forward Fluctuations in Noise-modulated Neural Networks

LocalProp: Neuro-Localized Memory-Efficient Backpropagation