Organizations: Department of Materials Science and Engineering, Delft University of Technology, Mekelweg 2, Delft, 2628CD, The Netherlands · Department of Precision and Microsystems Engineering, Delft University of Technology, Mekelweg 2, Delft, 2628CD, The Netherlands · Department of Intelligent Systems, Delft University of Technology, Van Mourik Broekmanweg 6, Delft 2628XE, The Netherlands
While backpropagation and automatic differentiation have driven deep learning's success, the physical limits of chip manufacturing and rising environmental costs of deep learning motivate alternative learning paradigms such as physical neural networks. However, most existing physical neural networks still rely on digital computing for training, largely because backpropagation and automatic differentiation are difficult to realize in physical systems. We introduce FFzero, a forward-only learning framework enabling stable neural network training without backpropagation or automatic differentiation. FFzero combines layer-wise local learning, prototype-based representations, and directional-derivative-based optimization through forward evaluations only. We show that local learning is effective under forward-only optimization, where backpropagation fails. FFzero generalizes to multilayer perceptron and convolutional neural networks across classification and regression. Using a simulated photonic neural network as an example, we demonstrate that FFzero provides a viable path toward backpropagation-free in-situ physical learning.
Figures & tables
Figure 1: Schematic comparison of FFzero and backpropagation performance when trained with directional derivatives. FFzero employs a local learning strategy in which each layer is updated independently, preventing gradient-estimation errors from compounding across layers. In contrast, backpropagation propagates directional-derivative estimation errors globally across layers, causing performance to degrade with increasing model size. FFzero maintains stable performance at scale while remaining compatible with in-situ physical implementation.
Figure 2: Schematic of FFzero algorithm, illustrated for a (a) 3-class classification problem and (b) regression problem. (a) Each linear layer is trained locally and independently by maximizing layer-wise goodness. The cosine similarities of the layer output w.r.t. the fixed prototype vectors (denoted by ξ[⋅] ) are used to compute the goodness. Directional derivatives are used to perform gradient-based optimization for each layer locally. (b) In regression tasks, the antipodal prototypes represent the upper ( y=1 ) and lower ( y=−1 ) prediction bounds, with intermediate ground-truth values encoded as interpolations between them. Each layer’s output is normalized onto a unit sphere and compared against the fixed antipodal vectors via cosine similarity, yielding layer-wise goodness for weights optimization via directional derivatives.
Figure 3: Schematic of directional derivative optimization in FFzero. (a) Illustration of directional derivative (DD) estimation on a representative goodness landscape. (b) Gradient estimation by averaging directional derivatives over multiple random directions. (c) Comparison of optimization trajectories obtained using DD-based updates and standard gradient ascent. While gradient ascent follows the true gradient direction, the expectation of the DD-based update matches the true gradient update (Supplementary Information 2). (d) Schematic illustration of the goodness-based DD update for a linear layer.
Figure 4: Classification accuracy of MLP models across varying hidden-layer widths, with all hidden layers constrained to equal size. Results are shown for (a) MNIST and (b) FashionMNIST datasets using backpropagation (BP) and forward–forward (FF) training paradigms, each combined with either automatic differentiation (AD) or directional-derivative (DD) optimization. The numbers above each plot indicate the total number of parameters for the corresponding model.
Figure 5: Classification accuracy of CNN models across varying number of channels per convolution layer. Results are shown for MNIST and FashionMNIST datasets using backpropagation (BP) and forward–forward (FF) training paradigms, each combined with either automatic differentiation (AD) or directional-derivative (DD) optimization. The numbers above each plot indicate the total number of parameters for the corresponding model.
Figure 6: Regression accuracy of MLP and CNN models. (a,b) Results are shown for regression on synthetic functions in equation ( 15 ) using MLPs across varying hidden-layer widths, with all hidden layers constrained to equal size. (c,d) Results are shown for regression on MNIST using MLP and CNN of varying hidden-layer widths and number of channels per convolution layer, respectively. For all cases, the comparison is made between backpropagation (BP) and forward–forward (FF) training paradigms, each combined with either automatic differentiation (AD) or directional-derivative (DD) optimization. The numbers above each plot indicate the total number of parameters for the corresponding model.
Figure 7: (a) Schematic of FFzero training applied to a two-layer (emulated) photonic neural network. (b) Test accuracy on MNIST using BP+AD, BP+DD, FF+AD, and FF+DD (FFzero).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Classification accuracy of MLP models across varying number of sampled directions used in directional-derivative (DD) optimization. Results are shown for MNIST dataset with MLPs of fixed depth d=5 and different hidden-layer widths: w∈{10,50,100} , with all hidden layers constrained to equal size. The results are compared between backpropagation (BP) and forward–forward (FF) training paradigms, each combined with either automatic differentiation (AD) or directional-derivative (DD) optimization.
Figure 2: PCA projection of the layer-wise embeddings of a two-layer photonic neural network trained with FFzero on MNIST. Each point represents one test sample, colored by the digit class. Stars denote the pre-computed prototype vectors projected into the same PCA space. After training with FFzero, data points cluster towards their corresponding prototype vectors in all layers.
Figure 3: Schematic illustration of the FFzero algorithm and architecture used for MLPs in ( a ) classification and ( b ) regression tasks.
Figure 4: Schematic illustration of the FFzero algorithm and architecture used for CNNs in ( a ) classification and ( b ) regression tasks. Within each convolution layer, feature maps from individual channels are reshaped and projected to a lower-dimensional space via fixed random linear mappings. The reduced representations are compared with fixed prototype vectors using cosine similarity, and a channel-wise goodness is computed to update the corresponding channel parameters through DD.
Parameter
Notation
Value
MLP for MNIST classification, MNIST regression,
and FashionMNIST classification:
Size of training dataset
-
50000
Size of test dataset
-
10000
Batch size
-
256
Number of training epochs
-
100
Appendix
Table 1: List of parameters and hyperparameters used in experiments.
Parameter
Notation
Value
CNN for FashionMNIST classification:
Size of training dataset
-
50000
Size of test dataset
-
10000
Number of training epochs
-
100
Learning rate
λ
0.001
Step size in DD
ϵ
0.001
Appendix
Table 1: (continued) List of parameters and hyperparameters used in experiments.
Training neural networks has traditionally relied on backpropagation (BP), a gradient-based algorithm that, despite its widespread success, suffers from key limitations in both biological and hardware perspectives. These include backward error propagation by symmetric weights, non-local credit assignment, and frozen activity during backward passes. We propose Forward Target Propagation (FTP), a biologically plausible and computationally efficient alternative that replaces the backward pass with a second forward pass. FTP estimates layerwise targets using only feedforward computations, eliminating the need for symmetric feedback weights or learnable inverse functions, hence enabling modular and local learning. We evaluate FTP on fully connected networks, CNNs, and RNNs, demonstrating accuracies competitive with BP on MNIST, CIFAR10, and CIFAR100, as well as effective modeling of long-term dependencies in sequential tasks. Moreover, FTP outperforms BP under quantized low-precision and emerging hardware constraints while also demonstrating substantial efficiency gains over other biologically inspired methods such as target propagation variants and forward-only learning algorithms. With its minimal computational overhead, forward-only nature, and hardware compatibility, FTP provides a promising direction for energy-efficient on-device learning and neuromorphic computing.
Nazmus Saadat As-Saquib, A N M Nafiz Abeer, Hung-Ta Chien +3
Texas A&M University, College Station, TX 77843 · Sandia National Laboratories, Livermore, CA 94550
A Noise-modulated Neural Network (NNN) learns and infers only in the presence of noise, treating noise as a computational resource rather than a disturbance. The noise lets it learn efficiently by backpropagation while transmitting spike-like signals, but backpropagation needs a reverse path through transposed weights, the weight transport problem, which undermines biological and neuromorphic plausibility. Forward-only alternatives typically substitute a different objective or fixed random feedback, sacrificing stability and accuracy. We show that backpropagation itself can be reconstructed in the NNN from forward-pass statistics alone: a weight mirror estimates each weight matrix from the covariance between a previous-layer unit's output and the next-layer unit's input, and combining it with local differential estimation inside the units propagates the output error recursively along the computational graph, with no transposed-weight readout and no backward data path. The resulting gradient is empirically near-unbiased, and with local per-weight Adam updates it matches the final accuracy of backpropagation on simple regression tasks. With uniformly distributed noise, the local operations reduce to polynomials and comparators, making the whole system, learning rule included, well suited to digital circuits. Thus, in the NNN, noise is a resource not only for inference but also for reconstructing backpropagation.
Shuhei Ikemoto
Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, Kitakyushu, Japan
The current deep learning training paradigm employs end-to-end backpropagation, regardless of the training stage, i.e. pre-training or fine-tuning. However, backpropagating through the entire model is neither biologically plausible nor memory efficient, since learning inside the brain is highly localized. Therefore, we propose LocalProp, a training procedure that locally updates the weights of a model. Our neuro-localized weight updates follow the "pre-training then fine-tuning" paradigm, where the pre-training is based on I-JEPA. After locally updating the weights, a pruning operation is performed, followed by a short final fine-tuning phase. Pruning helps by sending the learning signal from higher blocks to lower blocks. We perform experiments on several datasets, including large-scale benchmarks such as ImageNet, and empirically show that LocalProp reaches good performance at a fraction of GPU peak memory. By varying the number of jointly optimized blocks, we identify gradient-propagation span as a practical control over the accuracy-memory trade-off.
Diana-Nicoleta Grigore, Iuliana Georgescu, Radu Tudor Ionescu
Department of Computer Science University of Bucharest, Romania