Nonlinear Direct Feedback Alignment for Scalable Backpropagation-Free Training

Wu Lin; Yan Fan; Chenxiang Ma; Yinglan Feng; Jia Wang; Ka-Chun Wong; Qiuzhen Lin

cs.LGOct 27, 2026

Nonlinear Direct Feedback Alignment for Scalable Backpropagation-Free Training

Authors: Wu Lin, Yan Fan, Chenxiang Ma, Yinglan Feng, Jia Wang, Ka-Chun Wong, Qiuzhen Lin

Organizations: College of Computer Science and Software Engineering, Shenzhen University · School of Artificial Intelligence, Tianjin University · School of Artificial Intelligence, Shenzhen University · Intelligent Computing Center, Shenzhen University ·

Abstract

Backpropagation (BP) remains the predominant algorithm for training deep neural networks, but it is biologically implausible and suffers from the backward locking problem, constraining parallelization across layers. Direct Feedback Alignment~(DFA), which propagates the global output error directly to each hidden layer via fixed feedback matrices, has emerged as a promising alternative for addressing these limitations. However, a significant performance gap persists between DFA and BP, which we attribute to the limited ability of fixed feedback matrices to capture the nonlinear transformation from hidden representations to the network output. To address this limitation, we propose Nonlinear Direct Feedback Alignment (NDFA), which constructs each hidden-layer feedback using a lightweight nonlinear surrogate subnetwork. To enhance the effectiveness of the feedback throughout training, we further propose a learning strategy to update the surrogate subnetwork, thereby better approximating the underlying nonlinear transformation. Extensive experiments across residual networks and vision transformers on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets demonstrate that NDFA consistently outperforms existing DFA-based methods and substantially narrows the performance gap to BP, while maintaining efficient parallel training. The code is available at https://github.com/YanFan321/NDFA.

Figures & tables

Figure 1: Comparison of (a) BP, (b) DFA, and (c) the proposed NDFA method. BP propagates errors sequentially from the top to bottom layers, leading to the backward locking problem. DFA directly projects the global output error

\delta_{L}

to each hidden layer

l

using fixed random feedback matrices

B_{l}

. In contrast, NDFA constructs representation-dependent feedback through learnable nonlinear surrogate subnetworks

G_{l}

Table 2

Method	$K=4$			$K=16$
	CIFAR-10	CIFAR-100	SVHN	CIFAR-10	CIFAR-100	SVHN
BP ( $K{=}1$ )	92.23 $\pm$ 0.22	67.98 $\pm$ 0.22	96.71 $\pm$ 0.09	92.23 $\pm$ 0.22	67.98 $\pm$ 0.22	96.71 $\pm$ 0.09
DFA	72.73 $\pm$ 0.31	27.17 $\pm$ 0.59	90.85 $\pm$ 0.33	60.62 $\pm$ 2.19	15.24 $\pm$ 0.46	80.93 $\pm$ 1.07
DKP	63.91 $\pm$ 0.56	21.84 $\pm$ 0.78	90.02 $\pm$ 0.70	51.04 $\pm$ 0.61	18.59 $\pm$ 0.77	65.92 $\pm$ 1.32
FDFA	80.63 $\pm$ 0.35	49.12 $\pm$ 0.63	93.25 $\pm$ 0.32	77.36 $\pm$ 0.23	48.53 $\pm$ 0.22	92.77 $\pm$ 0.83
NDFA	85.36 $\pm$ 0.48	55.38 $\pm$ 0.38	94.82 $\pm$ 0.04	81.53 $\pm$ 0.60	50.45 $\pm$ 0.25	93.42 $\pm$ 0.19

Table 3: Comparison of different methods using ResNet-32.

K

is the number of parallel modules, where

K=16

represents the layer-wise setting.

Table 4

Figure 2: Comparison of feedback-related analyses between NDFA and other DFA-based methods on CIFAR-10 using ConvNet and AlexNet. Panels (a,e), (b,f), (c,g), and (d,h) show weight alignment, gradient alignment, subnetwork alignment, and surrogate subnetwork accuracy, respectively.

Figure 3: Comparison of layer-wise linear separability among different methods using AlexNet and ResNet-32 on CIFAR-10.

Table 7

Figure 4: Comparison of neural data similarity between BPTT and NDFA on the Mante 2013 dataset [ 41 ] .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture	Method	Learning Rate	Optimizer	Batch Size	Epochs
ConvNet	BP	5e-4	AdamW	256	100
	DFA	5e-4	AdamW	256	100
	DKP	8e-4	AdamW	256	100
	PEPITA	5e-2	SGD	256	100
	FDFA	1e-4	AdamW	256	100
	NDFA	1e-3	AdamW	256	100

Appendix

Table 8: Training hyperparameters used for different methods and architectures.

Method	FordA Acc. ( $\uparrow$ )	WikiText-2 PPL ( $\downarrow$ )
BP	$83.46\pm 0.31$	$84.02\pm 0.58$
DFA	$72.18\pm 0.74$	$229.64\pm 4.27$
NDFA	$\mathbf{82.71\pm 0.45}$	$\mathbf{87.96\pm 1.17}$

Appendix

Table 9: Results on FordA and WikiText-2 using ViT-Tiny.

Method	Accuracy (%)
Forward-Forward	$27.68\pm 0.21$
Forward-Forward ∗	$59.00$
LG-FG-A ∗	$69.32$
DNI	$62.68\pm 0.10$
DNI ∗	$64.86$
PredSim	$76.08\pm 0.18$

Appendix

Table 10: Comparison with different BP-free methods on CIFAR-10 using ConvNet. Results marked with

*

are copied from their original papers.

Figure 5: Comparison of layer-wise Jacobian alignment with BP for NDFA and DFA on CIFAR-10 using a ConvNet.

Loss	Epoch 1	Epoch 30	Epoch 50	Epoch 80	Epoch 100
Main Network	$1.69\pm 0.02$	$0.74\pm 0.02$	$0.66\pm 0.02$	$0.58\pm 0.02$	$0.57\pm 0.02$
1st Surrogate	$3.64\pm 0.07$	$2.26\pm 0.06$	$2.10\pm 0.05$	$1.98\pm 0.05$	$1.96\pm 0.05$
2nd Surrogate	$3.51\pm 0.04$	$1.81\pm 0.04$	$1.63\pm 0.04$	$1.48\pm 0.04$	$1.46\pm 0.04$
3rd Surrogate	$3.42\pm 0.04$	$1.47\pm 0.04$	$1.28\pm 0.04$	$1.11\pm 0.03$	$1.08\pm 0.03$

Appendix

Table 11: Convergence analysis on CIFAR-10 using ConvNet.

LR of Main Net	LR of Surrogate	Accuracy ( $\%$ )
$1\times 10^{-3}$	$1\times 10^{-3}$	$78.62\pm 0.46$
$1\times 10^{-3}$	$3\times 10^{-3}$	$78.74\pm 0.47$
$1\times 10^{-3}$	$5\times 10^{-3}$	$\mathbf{79.34\pm 0.50}$
$1\times 10^{-3}$	$5\times 10^{-4}$	$78.58\pm 0.42$
$1\times 10^{-3}$	$3\times 10^{-4}$	$78.45\pm 0.45$

Appendix

Table 12: Results of learning rate (LR) on CIFAR-10 using ConvNet.

Network	Method	CIFAR-10	CIFAR-100
ConvNet	Fixed NDFA w/ Linear Surrogate Subnets	$73.31\pm 0.26$	$40.29\pm 0.35$
	Fixed NDFA	$74.60\pm 0.09$	$41.65\pm 0.15$
	Learnable NDFA w/ Linear Surrogate Subnets	$76.23\pm 0.34$	$43.42\pm 0.58$
	NDFA	$\mathbf{78.62\pm 0.46}$	$\mathbf{47.86\pm 0.54}$
ResNet-32	Fixed NDFA w/ Linear Surrogate Subnets	$73.88\pm 0.32$	$37.79\pm 0.37$
	Fixed NDFA	$76.04\pm 0.38$	$40.10\pm 0.86$

Appendix

Table 13: Effect of nonlinearity in the surrogate subnetworks.

Architecture	Dimension=128			Surrogate Subnetwork Depth=2
Architecture	Depth	CIFAR-10	CIFAR-100	Dimension	CIFAR-10	CIFAR-100
ConvNet	2	78.62 $\pm$ 0.46	47.86 $\pm$ 0.54	128	78.62 $\pm$ 0.46	47.86 $\pm$ 0.54
	3	81.30 $\pm$ 0.28	49.04 $\pm$ 0.44	256	79.48 $\pm$ 0.23	48.16 $\pm$ 0.47
	4	82.95 $\pm$ 0.22	51.11 $\pm$ 0.49	512	79.82 $\pm$ 0.26	48.25 $\pm$ 0.38
ResNet-32	2	85.36 $\pm$ 0.48	55.38 $\pm$ 0.38	128	85.36 $\pm$ 0.48	55.38 $\pm$ 0.38
	3	86.76 $\pm$ 0.15	58.17 $\pm$ 0.29	256	85.69 $\pm$ 0.18	55.78 $\pm$ 0.35

Appendix

Table 14: Effects of surrogate subnetwork depth and hidden dimension.

Method	AlexNet	ResNet-32
BP	2.34 (1.00 $\times$ )	3.45 (1.00 $\times$ )
NDFA ( $d=2$ )	0.92 (2.54 $\times$ )	0.78 (4.42 $\times$ )
NDFA ( $d=3$ )	2.23 (1.05 $\times$ )	2.90 (1.19 $\times$ )
NDFA ( $d=4$ )	3.29 (0.71 $\times$ )	4.87 (0.71 $\times$ )

Appendix

Table 15: Per-iteration backward computation time (ms) of NDFA with different surrogate subnetwork depths

d

on CIFAR-10 using AlexNet and ResNet-32. Values in parentheses indicate the speedup relative to BP.

Depths (early $\to$ late)	Acc. ( $\%$ )	Time (ms)	FLOPs (G)	Params (M)
$2\to 2\to 2\to 2\to 2\to 2$	$87.38\pm 0.14$	0.92 (2.54 $\times$ )	140.30	45.82
$3\to 3\to 2\to 2\to 1\to 1$	$88.32\pm 0.34$	1.63 (1.44 $\times$ )	173.08	43.93
$3\to 3\to 2\to 2\to 2\to 2$	$88.88\pm 0.19$	1.69 (1.38 $\times$ )	175.06	44.90
$4\to 4\to 3\to 3\to 2\to 2$	$89.95\pm 0.16$	1.90 (1.23 $\times$ )	242.04	42.36
$6\to 5\to 4\to 3\to 2\to 1$	$\mathbf{90.54\pm 0.15}$	2.13 (1.10 $\times$ )	277.15	39.65
$2\to 2\to 2\to 2\to 3\to 3$	$87.43\pm 0.20$	1.52 (1.54 $\times$ )	209.56	79.39

Appendix

Table 16: Results of surrogate subnetworks with layer-dependent capacities using AlexNet on CIFAR-10. The surrogate depths are listed from the earliest to the latest network layer. Time is per-iteration backward time, and values in parentheses report speedup relative to BP.

Network	BP/NDFA Time (ms)	BP/NDFA FLOPs (G)	BP/NDFA GPU Memory (MB)
C2F2	7.29 / 6.89	5.25 / 6.04	263 / 263
C3F2	10.42 / 9.37	13.33 / 14.26	455 / 456
C5F2	12.31 / 14.05	21.35 / 22.65	454 / 455
C7F2	15.90 / 18.29	22.91 / 24.65	454 / 455
AlexNet	13.64 / 18.35	137.07 / 140.31	1894 / 1509
ResNet-32	37.34 / 46.88	35.39 / 38.80	880 / 831

Appendix

Table 17: Comparison of computational overhead on CIFAR-10.

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 332 papers

Jun 9, 2026

Overcoming Rank Collapse in Feedback Alignment

Backpropagation (BP) is widely viewed as biologically implausible, in part because it requires feedback weights to be the transpose of forward weights for error propagation. Interestingly, when training a network with fixed random feedback weights to circumvent this issue, learning aligns the forward weights with the feedback weights, leading the backpropagated error signal to become an approximation of the standard gradient used by BP. This process, called Feedback Alignment (FA), occurs in MLPs and very shallow CNNs but does not scale well to deeper architectures. In this work, we first investigated differences between BP and FA models, trained on CIFAR10, specifically focusing on the effective rank of the signal. We found that the FA error has a considerably lower rank and hence is constrained to a lower-dimensional subspace compared to BP, limiting exploration of the parameter space. Motivated by this observation, we evaluated two mechanisms for increasing the effective dimensionality of FA: Muon, an optimiser that orthogonalises weight updates; and hidden activity normalisation, which promotes activation orthogonality. Across larger architectures and benchmarks, we find that these methods consistently improve over FA baselines, for example, on CIFAR100 with a Resnet-18, accuracy increases by 9 percentage points. Our results identify low-dimensional gradient dynamics as a key obstacle to scaling FA and suggest that inducing higher-dimensional update geometry is a promising route toward scaling alternatives to backpropagation.

Gauthier Boeshertz, Razvan Pascanu, Claudia Clopath

Department of Bioengineering Imperial College London · Mila

Backpropagation Neural Network Optimization Neural Network Training Dynamics

Ask PDF

May 8, 2026

Biological Plausibility and Representational Alignment of Feedback Alignment in Convolutional Networks

The feedback alignment (FA) algorithm offers a biologically plausible alternative to backpropagation (BP) for training neural networks yet notably fails to scale to convolutional architectures. Modifications have been proposed to address this limitation, but at questionable cost to biological plausibility. In this paper, we evaluate five learning algorithms including modified FA and standard BP, applied to the same convolutional architecture with the CIFAR-10 dataset. We provide a tripartite comparative analysis focusing on biological plausibility, interpretability, and computational complexity. Our results indicate that modified FA algorithms converge on internal representations that are structurally similar to those produced by backpropagation. In particular, it appears the functional success of modified FA algorithms may be rooted in their ability to mimic the representational geometry of backpropagation, converging on similar representations despite relying on fundamentally different weight update mechanisms.

Jake Lance, Larry Kieu

University of Toronto

Representational Similarity Analysis Backpropagation Neural Network Interpretability Convolutional Neural Networks Biologically Plausible Learning

Ask PDF

Jul 20, 2026

Conditioned Direct Feedback Alignment via Activity and Error Geometry

Direct feedback alignment (DFA) trains hidden layers with fixed random projections of the output error, avoiding the transposed-weight backward pass of backpropagation (BP). We study a failure mode of DFA training that is distinct from feedback quality: the local weight update is calculated by an outer product, so anisotropy can enter through either its presynaptic-activity factor or its local-error factor. Our analyses with controlled synthetic regimes isolate the first failure mode and show an approximately 40-percentage-point activity-conditioning gain when high-variance directions contain task-irrelevant nuisance. Three clean confirmations isolate a different regime: error conditioning improves raw DFA by 1.77--7.53 percentage points, and combining independently selected activity and error factors adds 0.40--0.90 points over activity conditioning. The signs hold for tanh/one-vs-rest MNIST and preregistered Fashion-MNIST, and replicate on eight fresh seeds in a ReLU/softmax MNIST model. This factorization yields a symmetric block-local family of normalized DFA (nDFA): activity nDFA right-preconditions by an inverse activity second moment, error nDFA left-preconditions by an inverse local-error second moment, and K-nDFA applies both factors with separately tuned damping. A linearized post-alignment calculation gives an exact input-side spectral identity and a Kronecker-factor motivation for the two-sided rule, whereas norm matching rules out a scalar step-size explanation. The error factor is fragile when under-damped, BatchNorm is a strong activity-side alternative, and convnet gains remain partial. We therefore frame conditioned DFA as a factor-level study of when local outer-product rules fail, not as a general replacement for BP or a solution to all-layer convolutional credit assignment.

Houman Safaai, Varun Reddy, Bernardo L. Sabatini

Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University · Department of Neurobiology, Howard Hughes Medical Institute, Harvard Medical School

Multilayer Perceptrons Neural Network Training Dynamics

Ask PDF

Abstract

Neural Network Optimization

Explore similar work

Overcoming Rank Collapse in Feedback Alignment

Biological Plausibility and Representational Alignment of Feedback Alignment in Convolutional Networks

Conditioned Direct Feedback Alignment via Activity and Error Geometry