Common-Mode Collapse and Recovery in Direct Feedback Alignment

Varun Reddy; Bernardo L. Sabatini; Houman Safaai

cs.LGSep 25, 2026

Common-Mode Collapse and Recovery in Direct Feedback Alignment

Authors: Varun Reddy, Bernardo L. Sabatini, Houman Safaai

Organizations: Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University · Department of Neurobiology, Howard Hughes Medical Institute, Harvard Medical School

Abstract

Direct feedback alignment (DFA) trains hidden layers through fixed random projections of output error. With tanh hidden units and independent sigmoid outputs, plain stochastic gradient descent can stall near the loss of a constant predictor of class frequencies. We trace this stall to the error's common mode, the component shared across inputs. An exact mean-covariance decomposition separates a rank-one update formed by the mean teaching signal and mean presynaptic activity. Its leading component drives tanh units toward saturation. At initialization, random feedback provides no systematic correction of the shared error on average; readout learning limits its duration. A reduced model initialized from the network, without fitted parameters, predicts the concentration of activation sensitivity across 48 settings. On MNIST, class decodability largely survives collapse, but readout learning remains slow at a fixed learning rate. Adam learns faster despite deeper collapse. Calibrating the baseline readout to the class prior suppresses collapse and speeds learning; weaker feedback trades less collapse for slower learning. Replacing errors by their signs sustains collapse; subtracting the signal's batch mean prevents sustained collapse and improves learning in the tested setting. Related effects occur in deeper and convolutional networks and on CIFAR-10, with severity and cost depending on the readout, optimizer and input statistics.

Figures & tables

Figure 1: Collapse and recovery. (a–c) Schematic: shared error shifts unit means toward tanh saturation (b: open, initial; filled, shifted), concentrating gate energy. More units participate during recovery and their identities change. Points and bars are illustrative; numbers identify units. The solid arrow represents the approximate collapse mechanism, the dashed arrow observed recovery beyond the model. (d) Probe loss for DFA and matched BP (same initialization and minibatches); the horizontal line is the class-prior loss and shading marks the mean detected plateau interval, repeated in e–g. (e) Mean pairwise hidden cosine and (f) gate participation per layer; colours denote layers, solid curves DFA and dashed curves BP. (g) Shared-error norm

\|\bar{e}\|

and root-mean-square input-dependent error, both measured on the fixed probe. Time is linear through step 150 and logarithmic thereafter. Curves average 15 DFA trajectories and five BP networks; bands in d and g show one standard deviation.

Figure 2: The model predicts early concentration and approximate exit times, but not participation recovery. (a–c) Model (dashed) and matched network (solid), both using initialization/feedback seeds

(0,0)

: loss, gate participation and mean-activity energy. Dotted vertical lines in a mark predicted onset and exit, the exit from the covariance extension of Section 3.4 . (d) Participation minima within the first 300 updates across 48 settings and three seed pairs; colours denote layers (L1–L6). Points with model values near one but low measured minima are two-class and squared-error settings, for which the model predicts almost no drive. (e) Paired onset and exit times; the open triangle gives a lower bound for an exit not reached within the model horizon. Diagonals in d,e indicate equality.

Figure 3: Scaling, learning cost and mean-channel controls. (a)

1-\max_{t}\cos_{L}

against

g\|\bar{e}_{0}\|

: smaller values indicate higher cosine. Gain and initial-prediction sweeps are coloured, other sweeps grey; the line fits the coloured points, counting their shared baseline once. (b) Deepest-layer participation minimum within 300 updates against measured transient dose; dashed line: large-dose prediction capped at one. (c) Steps to 50% probe accuracy (filled; median) and to a sustained loss below

0.9\mathcal{L}_{\rm prior}

(open; mean) for the controls in d, centered output error and prior-bias initialization; dotted line: BP at 50% accuracy. (d) Peak cosine in each layer over the first 800 updates: input centering and frozen hidden biases jointly prevent collapse. Error bars in d: one standard deviation; trajectories in Figure 5 a–d.

Figure 4: Decodability, readout learning and useful descent. (a) Least-squares ridge decoding of one-hot targets on separate evaluation examples at three frozen layer-3 checkpoints. (b) Sigmoid/BCE readout continuation from the collapse checkpoint at nominal SGD rate $10^{-3}$ with raw, centered and centered/rescaled features; transformed parameters preserve the initial predictions. Grey curves continue the same raw (dashed) and centered, rescaled (dotted) readouts from the initial checkpoint. Curves and error bars in a,b show means and one standard deviation over three seed pairs. (c) Baseline gradient cosine against participation across 15 trajectories: grey curves are individual runs, coloured points follow the mean in time order; markers denote initialization, step 100 and step 3000.

Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$C,\;L,\;d_{\ell}$	Number of outputs, hidden layers and units in layer $\ell$ ; $i$ indexes units.
$x,\;h_{\ell},\;a_{\ell}$	Input, layer activity vector and preactivation vector; $h_{0}=x$ .
$z,\;\pi,\;\mathcal{L}_{\rm prior}$	Readout logits, class-frequency vector and loss of the constant prior predictor.
$\hat{y},\;y,\;e$	Prediction, target and output error $e=\hat{y}-y$ .
$\bar{e},\;\tilde{e},\;\hat{e}_{0}$	Mean error, deviation $e-\bar{e}$ and unit vector along the initial mean error.
$W_{\ell},\;b_{\ell},\;B_{\ell}$	Forward weights, biases and fixed feedback matrix.

Appendix

Table 1: Notation used in the analysis. A bar denotes an average over inputs, except for

\bar{u}_{\ell}

and

\overline{u^{2}}

, which average gate energies across units.

Diagnostic	Measures	Does not by itself establish
Hidden cosine	Similarity of activity directions	Equal magnitudes or loss of decodability
Gate participation $p_{\ell}$	Distribution of squared derivatives across units	Total gate strength or descent
Mean gate energy $\bar{u}_{\ell}$	Average squared derivative	Feedback magnitude or alignment
Error norm $\\|\bar{e}\\|$	Shared output teaching signal	Delivered parameter displacement
Gradient cosine $\cos\alpha_{\ell}$	Direction of the gradient estimate relative to BP	Size of loss improvement
Projected descent $\Pi_{\ell}$	Alignment weighted by relative magnitude	Optimizer-preconditioned descent

Appendix

Table 2: The diagnostics answer different questions. A single value does not establish the properties in the final column.

Intervention	Extra operations	Persistent state	Wall clock
Center output error	$O(nC)$	none	0.95
Center teaching signal	$O(nd)$ per layer	none	1.08
Prior bias / faster readout	none during training	none	0.92 / 0.98
Batch normalization	$O(nd)$ per layer	$O(d)$ per layer	1.10
Orthogonalize updates	$O(d^{3})$ per layer	none beyond SGD	1.21
Local-error conditioner	$O(nd^{2}+d^{3})$ per layer	$O(d^{2})$ per layer	1.74

Appendix

Table 3: Additional per-step cost for minibatch size

n

C

outputs and square hidden maps of width

d

, and descriptive mean wall clock per run relative to plain DFA on the base network. Rectangular orthogonalization costs depend on both matrix dimensions. Costs exclude the ordinary DFA update.

Figure 5: Causal dissection. (a–d) Input centering

\times

frozen hidden biases: the combined intervention prevents collapse at every layer without forcing all hidden means to zero. (e) Output error centered before projection. (f) DFA with feedback pre-aligned to the transposed downstream map at the Frobenius norm of the random draw. (g) BP with its forward readout weights scaled by 12, which also raises

\|\bar{e}_{0}\|

to 1.50: neither alignment control collapses. (h) Unit mean changes over the first 120 steps versus

(B_{\ell}\mathbf{1})_{i}

, for one baseline trajectory (seeds 0,0). Colours denote layers. Curves in a–g average runs; bands show one standard deviation. Counts are in Table 4 .

Condition	$n$	Peak $\cos_{3}$	Min. $p_{3}$	Accuracy
Plain DFA	15	$0.998\pm 0.000$	$0.225\pm 0.029$	$0.894\pm 0.006$
Backpropagation	5	$0.731\pm 0.012$	$0.951\pm 0.004$	$0.861\pm 0.006$
Layerwise FA	15	$0.991\pm 0.002$	$0.342\pm 0.039$	$0.882\pm 0.008$
Pre-aligned DFA	15	$0.772\pm 0.015$	$0.882\pm 0.007$	$0.912\pm 0.004$
BP, output weights $\times 12$	5	$0.607\pm 0.015$	$0.986\pm 0.002$	$0.927\pm 0.004$
Center output error	15	$0.379\pm 0.017$	$0.932\pm 0.011$	$0.877\pm 0.007$

Appendix

Table 4: MNIST baseline and mean-drive controls. Peak cosine uses the recorded trajectory; participation uses the first 300 updates. Accuracy is measured at step 3000. The frozen-readout control ends at step 1500, so its endpoint is unavailable (–). Softmax rows use globally standardized inputs, while other rows use raw pixels unless input centering is specified. Values are means

\pm

standard deviations.

Condition	$n$	Peak $\cos_{3}$	Learning time	50% acc.	80% acc.	Accuracy
Plain DFA	15	$0.998\pm 0.000$	$531\pm 34$	690	1220	$0.894\pm 0.006$
Frozen hidden biases	15	$0.998\pm 0.001$	$528\pm 35$	680	1220	$0.894\pm 0.006$
Centered inputs	15	$0.867\pm 0.024$	$345\pm 35$	430	910	$0.897\pm 0.006$
Centered inputs + frozen biases	15	$0.081\pm 0.011$	$1153\pm 30$	370	1050	$0.893\pm 0.007$
Centered output error	15	$0.379\pm 0.017$	$1898\pm 153$	370	1250	$0.877\pm 0.007$
Prior-bias initialization	15	$0.474\pm 0.145$	$288\pm 35$	420	860	$0.897\pm 0.006$

Appendix

Table 5: Learning cost of collapse in the MNIST baseline cohort. Learning time is the first probe loss below 90% of the constant-predictor loss that stays below it for 50 updates (the criterion of Table 21 ); mean

\pm

standard deviation. Accuracy columns give the median first step reaching 50% and 80% probe accuracy; the last column is accuracy at step 3000. Peak cosine uses the full trajectory, as in Table 4 . Centering prevents collapse and speeds discrimination but lowers mean-activity energy, which slows the readout’s fit to the class prior; the sigmoid loss then stays high although accuracy rises.

Layer	Trajectories	Spearman correlation
1	15	$-0.92\pm 0.03$
2	15	$-0.86\pm 0.03$
3	15	$-0.83\pm 0.04$

Appendix

Table 6: Unit mean shifts from initialization to step 120 against the shared-error feedback coefficient. Values are means and standard deviations of per-run Spearman correlations across the 15 baseline trajectories (five initializations crossed with three feedback draws); they are descriptive spreads, not independent-run confidence intervals.

Condition	$n$	Min. variance / initial	Initial rank	Min. rank
Plain DFA	15	$0.028\pm 0.007$	$45.6\pm 0.7$	$4.3\pm 0.3$
Backpropagation	5	$0.826\pm 0.008$	$45.6\pm 0.7$	$37.4\pm 0.6$
Pre-aligned DFA	15	$0.808\pm 0.013$	$45.6\pm 0.7$	$12.2\pm 0.2$
Prior-bias initialization	15	$0.967\pm 0.126$	$45.6\pm 0.7$	$5.3\pm 0.2$
Centered inputs + frozen biases	15	$1.000\pm 0.000$	$43.7\pm 0.6$	$4.8\pm 0.8$
Centered output error	15	$1.000\pm 0.000$	$45.6\pm 0.7$	$4.6\pm 0.6$

Appendix

Table 7: Centered geometry of top-layer activity in the MNIST baseline cohort. The centered variance is

\mathbb{E}_{x}\|h_{3}-\bar{h}_{3}\|^{2}

, the amplitude of activity differences across inputs, and its minimum within the first 500 updates is given relative to initialization. The centered effective rank is the exponential of the entropy of the normalized centered covariance spectrum; its minimum uses the same 500-update window. Means

\pm

standard deviations across runs.

$\eta$	Rule	$n$	Min. $p_{3}$	50% acc.	80% acc.	Accuracy
$0.001$	DFA	3	$0.2143\pm 0.0191$	690	1150	$0.902\pm 0.005$
$0.001$	DFA, prior bias	3	$0.8545\pm 0.1410$	420	800	$0.904\pm 0.003$
$0.001$	BP	3	$0.9504\pm 0.0054$	210	1040	$0.863\pm 0.006$
$0.003$	DFA	3	$0.2053\pm 0.0203$	236	406	$0.925\pm 0.005$
$0.003$	DFA, prior bias	3	$0.8471\pm 0.1342$	138	270	$0.928\pm 0.004$
$0.003$	BP	3	$0.9481\pm 0.0058$	66	324	$0.904\pm 0.004$

Appendix

Table 8: Plain-SGD learning rate in the MNIST baseline, with equal hidden and readout rates. Columns give diverged runs (non-finite loss, or final loss above the initial loss), the participation minimum within 300 updates, the median first update reaching 50% and 80% probe accuracy, and accuracy after 3,000 updates, over non-diverged runs with seed pairs

(0,0)

(1,1)

and

(2,2)

. The

10^{-3}

rows reuse the headline runs with initialization seeds 0–2 (BP feedback seeds are unused). Prior bias sets all sigmoid biases to

\operatorname{logit}(1/C)

. Probes are logged every ten updates at

10^{-3}

and every two at larger rates. Each new run completes 3,000 updates.

$C$	$n$	Ideal $\\|\bar{e}_{0}\\|$	Measured $\\|\bar{e}_{0}\\|$
2	3	$0.000$	$0.057\pm 0.013$
3	3	$0.289$	$0.266\pm 0.088$
5	3	$0.671$	$0.700\pm 0.083$
10	3	$1.265$	$1.289\pm 0.055$

Appendix

Table 9: Initial mean error under sigmoid outputs as the number of MNIST classes changes. The ideal value assumes a constant prediction of one half. The measured value uses a random readout. Class restriction also changes training-set size; this comparison tests initialization, not later learning speed.

Setting	Step	Layer 1	Layer 2	Layer 3
MNIST tanh	0	$0.01\pm 0.00$	$0.01\pm 0.00$	$0.01\pm 0.00$
	50	$0.22\pm 0.02$	$0.23\pm 0.02$	$0.11\pm 0.03$
	100	$1.00\pm 0.10$	$0.83\pm 0.09$	$0.37\pm 0.09$
	300	$1.89\pm 0.93$	$1.57\pm 0.82$	$1.48\pm 0.89$
MNIST ReLU	0	$0.07\pm 0.00$	$0.06\pm 0.01$	$0.05\pm 0.00$
	50	$0.94\pm 0.13$	$0.42\pm 0.05$	$0.19\pm 0.07$

Appendix

Table 10: Gate–error residual norm relative to the leading mean-teaching norm, mean

\pm

standard deviation over three seed pairs. Absolute norms are retained in the supplementary data.

Figure 6: Checks of the collapse and recovery approximations. (a,b) Signed contributions along the measured virtual displacement of hidden layer-3 means and readout mean logits; finite cross terms are retained in the data. (c) Measured mean-error norm and the discrepancy from applying the sigmoid to the mean logit. (d) Layer-3 covariance energy in the network and the four recovery approximations defined in Appendix C.5 (legend below). (e) Absolute norms of the hidden layer-3 drift contributions in a. (f) Norm ratio of the covariance and rank-one contributions of shared output error in Equation ( 11 ). All panels use three baseline seed pairs; curves show means, and bands where shown one standard deviation. These are snapshot checks and model comparisons, not independent causal interventions.

Location	Step	Mean term	Upstream	Covariance	$\\|\Delta\mu\\|$ or $\\|\Delta\bar{z}\\|$
Hidden layer 3	0	$0.15\pm 0.01$	$0.85\pm 0.01$	$-0.00\pm 0.00$	$0.6715\pm 0.0644$
Hidden layer 3	50	$0.85\pm 0.00$	$0.15\pm 0.01$	$0.01\pm 0.01$	$0.2405\pm 0.0426$
Hidden layer 3	100	$0.69\pm 0.08$	$0.22\pm 0.05$	$0.05\pm 0.04$	$0.0837\pm 0.0187$
Hidden layer 3	300	$0.30\pm 0.38$	$0.35\pm 0.15$	$-0.02\pm 0.05$	$0.1083\pm 0.0242$
Readout	0	$-0.02\pm 0.05$	$1.01\pm 0.05$	$-0.00\pm 0.00$	$0.1611\pm 0.0378$
Readout	50	$0.94\pm 0.02$	$0.05\pm 0.02$	$0.00\pm 0.00$	$0.0778\pm 0.0082$

Appendix

Table 11: Signed projections onto the measured virtual mean displacement, divided by its squared norm. The complete components sum to one; the displayed columns omit the gate residual or output-map error and the finite cross term, so they need not sum to one. Absolute displacement norms are shown because large fractions can arise when net motion is small. Means and standard deviations use three seeds.

Summary	Conditions	Points	$r$	95% interval	MAE
Pooled	48	450	0.89	$[0.81,0.94]$	0.105
Condition means	48	48	0.91	$[0.83,0.97]$	0.101
Layer 1	48	144	0.93	$[0.87,0.96]$	0.128
Layer 2	48	144	0.90	$[0.82,0.95]$	0.112
Layer 3	48	144	0.86	$[0.73,0.95]$	0.087
Layer 4	2	6	-0.16	–	0.027

Appendix

Table 12: Participation prediction across distinct physical conditions. Intervals resample whole conditions, retaining their seed and layer observations (2,000 bootstrap draws). Condition means average over seeds and layers; deeper layers contain only two conditions and have no reported interval.

Settings	Predictor	Points	$r$	MAE	$r$ , means	MAE, means
All 48 settings	Reduced model	450	0.89	0.105	0.91	0.101
	Closed-form estimate	450	0.39	0.333	0.64	0.307
37 tanh, sigmoid, $C>2$ settings	Reduced model	351	0.96	0.077	0.99	0.072
	Closed-form estimate	351	0.43	0.318	0.67	0.293

Appendix

Table 13: Participation minima predicted by the reduced model and by the closed-form initialization estimate,

\min(1,1.5/\hat{\kappa}_{\ell})

with

\hat{\kappa}_{\ell}

from Equation ( 6 ), on condition–seed–layer points. The closed form uses the sigmoid effective slope; the second block restricts both predictors to the settings its derivation covers. Condition means average seeds and layers, as in Table 12 . Within single-parameter sweeps, the reduced-model correlation over setting-by-layer means is: gain 0.98, output bias 0.99, learning rate 0.99, width 0.98, batch size 0.99, class count 0.10; the class-count sweep includes the two-class setting, where the model predicts almost no drive.

Comparison	Runs	Both	Network only	Model only	Neither	Exit pairs	Censored
Parameter sweeps	144	77	1	13	53	76	1
Width–depth grid	48	39	4	3	2	39	0

Appendix

Table 14: Plateau detection and timing coverage. Both, network only, model only and neither partition all trajectories by detected onset. Censored model exits are included in Both but excluded from numerical exit errors. Plateau detection uses loss, not hidden-state collapse.

Setting	Exit: 1	2	4	Max. $\|\Delta\min p_{\ell}\|$
Baseline	570	567	565	0.0053
Gain 10	213	209	208	0.0054
Width 100, depth 7	2809	2801	2797	0.0023

Appendix

Table 15: Numerical resolution in three selected settings, each with initialization/feedback seeds

(0,0)

. Exit is in update steps for one, two or four Euler substeps. The last column is the largest absolute change in the first-300-update participation minimum across layers from one to four substeps. These use 64 quadrature nodes; orders 32 and 128 give the same displayed minima and timing. All detailed values and initial states accompany the code.

Figure 7: Width and depth test the reduced model jointly. (a,b) Deepest-layer participation minima within the first 300 updates in the network and model, averaged over three seeds per cell. (c) Mean absolute error, computed per seed before averaging. (d,e) Plateau exit in update steps, averaged over observed exits; a cross marks a cell with no detected plateau, and an open triangle a cell with a detected plateau but no exit within the horizon. Table 14 separates detection failures from censored exits; per-cell counts are saved with the comparison. (f) A separate three-layer sweep with widths 100, 300, 900 and 2000: layer-1 and top-layer participation minima, and plateau exit on the right axis (orange triangles; dashed line: power-law fit).

Group / variant	Runs	Both	Missed	Extra	Exit pairs	Median rel. error
fresh / full	9	9	0	0	9/9	0.19
fresh / no local	9	9	0	0	0/9	–
fresh / no participation	9	9	0	0	9/9	0.96
fresh / measured init	9	9	0	0	9/9	0.20
grid / full	48	39	4	3	39/43	0.24
grid / no local	48	43	0	5	7/43	1.77

Appendix

Table 16: Recovery alternatives at each network’s recorded training horizon. Both, missed and extra refer to plateau onset detection. Exit pairs count observed model/network exits relative to all observed network exits; timing error is conditional on paired exits. Each variant uses the same collapse trajectory and detector. The fresh group uses new width/depth combinations and seeds, with predictions saved before training.

Predictor	Exponent	95% interval
$a=\eta_{\rm hid}g$	-0.40	[-0.42, -0.38]
$\eta_{\rm out}$	-0.60	[-0.62, -0.57]

Appendix

Table 17: Two-rate recovery fit after merging equivalent SGD gain/rate settings. The fit is conditional on observed plateau durations; intervals resample whole effective-rate conditions (2,000 draws).

Checkpoint	Layer	Covariance energy	$R$	Feature RMS	Decoder accuracy
initial	1	$0.472\pm 0.012$	$5.99\pm 0.01$	$0.274\pm 0.002$	$0.877\pm 0.007$
initial	2	$0.372\pm 0.029$	$6.14\pm 0.03$	$0.244\pm 0.003$	$0.890\pm 0.006$
initial	3	$0.300\pm 0.016$	$6.21\pm 0.02$	$0.222\pm 0.003$	$0.893\pm 0.007$
collapse	1	$0.440\pm 0.026$	$6.37\pm 0.05$	$0.227\pm 0.003$	$0.892\pm 0.006$
collapse	2	$0.103\pm 0.020$	$6.31\pm 0.06$	$0.105\pm 0.007$	$0.896\pm 0.005$
collapse	3	$0.014\pm 0.005$	$5.94\pm 0.11$	$0.038\pm 0.005$	$0.886\pm 0.005$

Appendix

Table 18: Frozen-feature diagnostics with 10,000 training examples for fitting and 5,000 separate validation examples for evaluation. A fixed trace-relative ridge of

10^{-3}

is used for least-squares one-hot decoding. Here

R

uses the fitting set and decoder ridge, whereas Table 19 uses the probe estimator. The collapse checkpoint is each run’s layer-3 participation minimum within its first 300 updates. Values average three seeds.

Condition	$n$	$t_{*}$	Energy ratio	$R_{3}(0)$	$R_{3}(t_{*})$
Plain DFA	3	$80\pm 0$	$28.0\pm 6.6$	$7.64\pm 0.03$	$7.42\pm 0.10$
Backpropagation	3	$147\pm 12$	$1.2\pm 0.0$	$7.64\pm 0.03$	$7.74\pm 0.00$
Pre-aligned DFA	3	$40\pm 0$	$0.9\pm 0.0$	$7.64\pm 0.03$	$7.86\pm 0.01$
Center teaching signal	3	$40\pm 0$	$0.8\pm 0.0$	$7.64\pm 0.03$	$7.69\pm 0.01$
Frozen readout	3	$253\pm 50$	$7785.5\pm 5851.3$	$7.64\pm 0.03$	$5.63\pm 0.26$
Feedback gain 3	3	$40\pm 0$	$187.4\pm 56.5$	$7.64\pm 0.03$	$7.01\pm 0.11$

Appendix

Table 19: Covariance amplitude and its normalized statistic at matched layer-3 checkpoints. Within each run,

t_{*}\geq 30

minimizes covariance energy among checkpoints with a recorded

R_{3}

. The energy ratio is

\|\Lambda_{3}(0)\|_{F}^{2}/\|\Lambda_{3}(t_{*})\|_{F}^{2}

. A large amplitude loss need not imply a comparable loss of normalized covariance.

R_{3}

is not held-out accuracy.

Figure 8: Unit means and spreads in the Gaussian reconstruction. Panels (a–c) show layers 1–3. Measured participation loss is compared with the full per-unit Gaussian reconstruction and two moment substitutions: one shared root-mean-square spread with measured unit means, or zero means with measured unit spreads. Curves average

-\log p_{\ell}

across 15 baseline trajectories at each saved snapshot. The horizontal axis is linear through step 100 and logarithmic thereafter. These substitutions change the reconstruction, not the network’s training.

Figure 9: Gate turnover and the direction and size of learning signals. (a) Jaccard overlap of the current top-fifth gate-energy set with its step-120 set; dashed line: chance overlap. (b) Layer-3 alignment of norm-matched virtual weight-and-bias updates from all units, current or fixed top-fifth sets, a random fifth, and the fixed set’s complement; time is linear through step 100 and logarithmic thereafter. (c,d) Layer-3 raw weight-gradient cosine and DFA-to-BP norm ratio for random and pre-aligned feedback. Curves and bands show means and one standard deviation: 15 trajectories in a,c,d and three paired seeds in b. The complement in b contains more units, but every virtual step has the same norm.

Figure 10: Interventions. All networks, including the CNN, are trained on MNIST. (a) Probe accuracy at 1,000, 1,500 and 3,000 steps on the base network. (b) Accuracy at 3,000 steps in the base and six-layer MLPs; CNN runs end earlier. (c) Minimum top-layer participation within 300 steps in both MLPs and the CNN. Points and bars show means and one standard deviation; crosses mark unmeasured conditions. (d) Accuracy at step 3000 and peak cosine versus relative ridge

\lambda_{\rm rel}

; dashed lines show plain DFA. The conditioner appears at its default ridge

0.1

and, for the base MLP, at the ridge

10

selected by the same step-3000 validation accuracy. The orthogonalize control uses Muon-style orthogonalization, not full Muon. All measured conditions use three paired seeds.

Protocol	$\eta_{\rm hid}$	ReLU loss	GELU loss	Linear loss
Plain DFA	$10^{-4}$	$2.9(0.3)\!\times 10^{0}$	$2.8(0.15)\!\times 10^{0}$	$1.8(0.083)\!\times 10^{0}$
Center error	$10^{-4}$	$3.2(0.23)\!\times 10^{0}$	$3.7(0.15)\!\times 10^{0}$	$1.4(1.3)\!\times 10^{1}$
Plain DFA	$3\times 10^{-4}$	$1(0.027)\!\times 10^{0}$	$1(0.041)\!\times 10^{0}$	$9.8(0.3)\!\times 10^{-1}$
Center error	$3\times 10^{-4}$	$1.5(1.3)\!\times 10^{2}$	$1.4(1.2)\!\times 10^{3}$	$7.4(9.3)\!\times 10^{10}$
Center error + faster readout	$3\times 10^{-4}$	$2.3(2)\!\times 10^{2}$	$2.4(1.9)\!\times 10^{3}$	$1.3(1)\!\times 10^{5}$
Backpropagation	$10^{-3}$	$1.4(0.058)\!\times 10^{0}$	$1.7(0.091)\!\times 10^{0}$	$9.9(0.19)\!\times 10^{-1}$

Appendix

Table 20: Final loss with non-saturating units after 3000 updates. All 27 activation–protocol combinations are retained, with three seeds each. Entries give the mean (standard deviation) in scientific notation. The hidden learning rate is shown; a faster readout uses ten times that rate. Large finite losses indicate instability and are not nonfinite early stops. The constant-predictor reference is about

3.2491

Figure 11: Adam produces deeper gate concentration with faster loss recovery. (a) Probe loss and the prior-loss reference. (b) Layer-3 participation. (c) Mean-error controls and BP at the larger Adam learning rate. (d) Layer-3 cosine between the actual displacement and negative true gradient, including hidden biases. Colours in b,d follow a; panel c has its own legend. Curves show means and bands show one standard deviation across three seed pairs. The SGD dose formula is not applied to Adam.

Condition	$n$	Time	Reached/ $n$	Final accuracy
Plain SGD	3	$537\pm 47$	3/3	$0.902\pm 0.005$
Adam, $\eta=10^{-3}$	3	$117\pm 6$	3/3	$0.969\pm 0.004$
Adam, $\eta=10^{-4}$	3	$480\pm 36$	3/3	$0.928\pm 0.006$
Conditioner, $\lambda_{\rm rel}=0.1$	3	$1897\pm 67$	3/3	$0.582\pm 0.044$
Conditioner, $\lambda_{\rm rel}=3$	3	$433\pm 21$	3/3	$0.866\pm 0.005$
Conditioner, $\lambda_{\rm rel}=10$	3	$387\pm 23$	3/3	$0.893\pm 0.007$

Appendix

Table 21: Learning time without requiring a detected plateau: first probe loss below 90% of the constant-predictor loss, remaining below it for 50 updates. All runs use the same criterion and seed pairs

(0,0)

(1,1)

and

(2,2)

; the shared SGD baseline is shown once, so its accuracy can differ slightly from the 15-trajectory value in Table 4 . Values are means

\pm

standard deviations; counts state how many reach it within 3,000 updates.

Protocol	Rate	$n$	Peak $\cos_{3}$	Saturated	Leaves chance	Learning time	80% acc.	Accuracy
DFA, zero initialization	$10^{-3}$	3	$1.000\pm 0.000$	$1.00\pm 0.00$	$40\pm 0$	$40\pm 0$	$60\pm 0$	$0.973\pm 0.005$
BP, fan-in initialization	$10^{-3}$	3	$1.000\pm 0.000$	$1.00\pm 0.00$	$223\pm 162$	$273\pm 189$	$467\pm 151$	$0.974\pm 0.004$
DFA, Xavier initialization	$10^{-3}$	3	$1.000\pm 0.000$	$1.00\pm 0.00$	$40\pm 0$	$40\pm 0$	$60\pm 0$	$0.972\pm 0.010$
DFA, zero initialization	$10^{-4}$	3	$1.000\pm 0.000$	$0.12\pm 0.01$	$47\pm 6$	$70\pm 0$	$187\pm 6$	$0.959\pm 0.002$
BP, fan-in initialization	$10^{-4}$	3	$0.954\pm 0.002$	$0.25\pm 0.00$	$10\pm 0$	$10\pm 0$	$43\pm 6$	$0.969\pm 0.003$
DFA, Xavier initialization	$10^{-4}$	3	$0.987\pm 0.004$	$0.29\pm 0.12$	$40\pm 0$	$80\pm 10$	$220\pm 10$	$0.967\pm 0.001$

Appendix

Table 22: Reconstruction of the original DFA protocol ( Nøkland, 2016 ) :

3\times 800

tanh MLP, logistic outputs with binary cross-entropy, inputs in

[0,1]

, RMSprop and minibatches of 64, with zero initialization and uniform feedback for DFA and fan-in initialization for BP; the last rows keep Xavier initialization and Gaussian feedback. Saturated is the largest fraction of top-layer units with

|\bar{h}_{3i}|>0.9

. Leaves chance is the first update with probe accuracy above the majority-class rate plus 0.05; learning time uses the criterion of Table 21 ; the 80% column is the first update reaching that accuracy; the last column is accuracy after 6,000 updates. At zero initialization, layers above the first are input-independent until their weights grow, so their peak cosine is one by construction. Means

\pm

standard deviations over three seed pairs.

Condition	$n$	Peak $\cos_{3}$	Final $\cos_{3}$	Min. $p_{3}$	Final $p_{3}$	Learning time	Accuracy
Sign	3	$1.000\pm 0.000$	$0.957\pm 0.010$	$0.013\pm 0.003$	$0.055\pm 0.008$	$1833\pm 398$	$0.750\pm 0.031$
Sign, prior bias	3	$1.000\pm 0.000$	$0.957\pm 0.010$	$0.013\pm 0.003$	$0.055\pm 0.008$	$1830\pm 393$	$0.750\pm 0.031$
Sign, centered	3	$0.378\pm 0.021$	$0.026\pm 0.001$	$0.892\pm 0.021$	$0.872\pm 0.009$	$1073\pm 31$	$0.896\pm 0.005$

Appendix

Table 23: Sign-error feedback in the MNIST baseline: hidden layers receive

\operatorname{sign}(e)

through the baseline feedback, and the readout keeps its gradient. Participation minima use the first 300 updates; final values and accuracy are at step 3000. Learning time uses the criterion of Table 21 . Plain sign feedback and prior-bias initialization give identical hidden updates, because

\operatorname{sign}(e)=\mathbf{1}-2y

does not depend on the readout. Means

\pm

standard deviations over three seed pairs.

Figure 12: Generality across protocols. Deepest-layer cosine under plain DFA (layerwise FA in c,d), centered error, and matched BP where available; each BP curve is a single run. Panel titles specify the protocols; MSE denotes squared error. Curves show run means; bands show one standard deviation for plain DFA/FA. ReLU and GELU cosines require complementary gate and loss measurements.

Readout and targets	$n$	Initial $\\|\bar{e}\\|$	Peak cosine	Min. $p_{3}$
Centered squared error, random readout	3	$0.803\pm 0.346$	$0.995\pm 0.002$	$0.366\pm 0.066$
Centered squared error, zero readout	3	$0.022\pm 0.006$	$0.378\pm 0.020$	$0.987\pm 0.001$
Same zero readout, BP	3	$0.022\pm 0.006$	$0.378\pm 0.021$	$0.994\pm 0.001$
Uncentered squared error, zero readout	3	$0.317\pm 0.000$	$0.752\pm 0.014$	$0.944\pm 0.007$
Sigmoid, zero readout	3	$1.265\pm 0.000$	$0.998\pm 0.000$	$0.243\pm 0.014$
Softmax, zero readout	3	$0.022\pm 0.006$	$0.250\pm 0.007$	$0.940\pm 0.010$

Appendix

Table 24: Readout initialization and target centering in the baseline architecture. Zero readout means zero output weights; sigmoid predictions are then one half at initialization. Peak cosine uses the 2000-update trajectory, and participation minima use the first 300 updates.

Figure 13: CIFAR-10. Hidden cosine of CNN block 3 (the 128-unit fully connected hidden layer; a–c) or MLP layer 1 (d) under DFA, the matched BP network and the interventions. (a) Small CNN with sigmoid readout, comparing prior-bias initialization, mean-logit calibration and centering controls. (b) CNN with balanced softmax and standardized inputs. (c) CNN with softmax and class-0 mixture weight

p_{0}=0.7

. (d) Flattened-input MLP with error centering, per-pixel centering, global standardization or mean-logit calibration. BP uses matched initialization and minibatches. Curves show means across three seed pairs, bands one standard deviation. High cosine alone does not establish collapse; Table 25 reports complementary measurements.

Condition	$n$	Peak cosine	Min. participation	Accuracy
CNN: plain DFA	3	$1.000\pm 0.000$	$0.009\pm 0.001$	$0.276\pm 0.021$
CNN: BP	3	$0.964\pm 0.008$	$0.756\pm 0.026$	$0.343\pm 0.006$
CNN: prior bias	3	$1.000\pm 0.000$	$0.016\pm 0.002$	$0.349\pm 0.039$
CNN: calibrated logits	3	$1.000\pm 0.000$	$0.016\pm 0.010$	$0.340\pm 0.015$
CNN: center error	3	$0.876\pm 0.013$	$0.905\pm 0.015$	$0.358\pm 0.031$
CNN: center error + inputs	3	$0.139\pm 0.030$	$0.850\pm 0.023$	$0.349\pm 0.019$

Appendix

Table 25: CIFAR-10 controls. Conditions shared with Figure 13 use the same runs and layers. CNN geometry uses hidden block 3, the 128-unit fully connected layer after two convolutional blocks; MLP geometry uses layer 1. Entries average per-run peak cosine over 4000 updates and minimum participation within 300 updates, with sample standard deviations. Accuracy is measured at step 3000. Softmax CNNs use globally standardized inputs. Other rows use raw pixels except the stated centering or standardization controls. Imbalanced accuracy should be read with Table 26 .

Dataset / rule	$n$	Majority reference	Accuracy	Balanced accuracy
CIFAR-10, DFA	3	$0.73\pm 0.00$	$0.74\pm 0.00$	$0.21\pm 0.03$
CIFAR-10, BP	3	$0.73\pm 0.00$	$0.73\pm 0.01$	$0.18\pm 0.02$
CIFAR-10, centered error	3	$0.73\pm 0.00$	$0.71\pm 0.00$	$0.21\pm 0.01$
MNIST, DFA	15	$0.73\pm 0.00$	$0.96\pm 0.01$	$0.88\pm 0.01$
MNIST, BP	5	$0.73\pm 0.00$	$0.95\pm 0.01$	$0.83\pm 0.01$

Appendix

Table 26: Imbalanced softmax controls at their final recorded update (MNIST: 3000; CIFAR-10: 4000). The majority-class reference uses the training sampling prior; balanced accuracy averages class recall on the fixed probe. Raw accuracy alone can be close to the majority predictor.

Condition	$n$	Shift	Cos. before	Cos. after	$\\|\bar{e}\\|$ : first	+50
DFA, early shift	3	300	$0.09\pm 0.07$	$0.06\pm 0.04$	$0.473\pm 0.033$	$0.082\pm 0.005$
BP, early shift	3	300	$0.22\pm 0.01$	$0.21\pm 0.01$	$0.414\pm 0.067$	$0.098\pm 0.014$
DFA, late shift	3	1500	$0.03\pm 0.01$	$0.03\pm 0.01$	$0.105\pm 0.019$	$0.035\pm 0.017$
BP, late shift	3	1500	$0.11\pm 0.00$	$0.11\pm 0.00$	$0.172\pm 0.025$	$0.069\pm 0.006$

Appendix

Table 27: Class-prior shifts in the MNIST softmax MLP described in Appendix E.6 . Peak cosine uses the 100 updates before and 300 updates after the shift. Error norms use the first shifted training batch and the batch 50 updates later. The fixed validation probe retains its pre-shift distribution.

Figure 14: Sampling noise and collapse are separate measurements. (a,b) Root-mean-square batch-mean error at frozen initialization: raw norm and fluctuation about the population mean, with the sampling identity shown dashed. (c,d) Peak top-layer cosine during 1,500 updates with the same readout choices. Panels a,c use softmax MNIST; b,d use a mean-logit-calibrated sigmoid CIFAR-10 MLP. Curves show means and bands or bars one standard deviation across three seeds. Changing batch size also changes the number of examples processed.

Protocol	$n$	Initial $\\|\bar{e}\\|$	Peak cosine	Min. $p_{3}$
Raw pixels, sigmoid	3	$1.289\pm 0.055$	$0.998\pm 0.000$	$0.214\pm 0.019$
Raw pixels, softmax	3	$0.078\pm 0.032$	$0.548\pm 0.237$	$0.920\pm 0.045$
Standardized, softmax	3	$0.091\pm 0.002$	$0.297\pm 0.024$	$0.930\pm 0.011$

Appendix

Table 28: MNIST readout comparisons with preprocessing stated explicitly. All rows use the same three initialization–feedback pairs,

(0,0)

(1,1)

and

(2,2)

, batch size 128, and a 1,500-update window for peak cosine; participation minima use the first 300 updates. Table 4 reports larger 15-trajectory cohorts for the corresponding headline protocols; the raw-pixel softmax row is an additional preprocessing-matched comparison. Only the first two rows isolate the readout choice.

Readout, rule	$n$	Initial $\\|\bar{e}\\|$	Peak cosine	Min. $p_{3}$	Accuracy
Sigmoid, DFA	3	$4.976\pm 0.015$	$1.000\pm 0.000$	$0.036\pm 0.007$	$0.011\pm 0.004$
Sigmoid, BP	3	$4.976\pm 0.015$	$0.999\pm 0.000$	$0.282\pm 0.006$	$0.018\pm 0.005$
Softmax, DFA	3	$0.046\pm 0.004$	$0.986\pm 0.011$	$0.632\pm 0.100$	$0.028\pm 0.004$
Softmax, BP	3	$0.046\pm 0.004$	$0.777\pm 0.020$	$0.954\pm 0.003$	$0.069\pm 0.007$

Appendix

Table 29: CIFAR-100 readouts in the three-layer tanh MLP under DFA and matched BP (same initializations and minibatches). High peak cosine also occurs under BP on these uncentered images; participation separates the rules, and sigmoid outputs concentrate gate energy most. Participation minima use the first 300 updates; peak cosine and final accuracy use the 3000-update run. Accuracy remains low for both rules after 3000 updates at this learning rate, so these runs test the early transient rather than final performance.

Figure 15: The teacher–student ladder. Test loss relative to the constant predictor (top), top-layer hidden cosine (middle) and DFA weight alignment (bottom) against

t=\mathrm{step}/D

, where

D

is input dimension. DFA is solid and BP dashed; thin lines show the three individual seeds. Column a adapts Refinetti et al. (2021) to four outputs; b–f change the objective and targets, hidden biases, depth and activation, input statistics, then feedback gain. Weight alignment is the cosine between vectorized feedback and downstream weight matrices (Appendix A.2 ).

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 520 papers

Jul 20, 2026

Conditioned Direct Feedback Alignment via Activity and Error Geometry

Direct feedback alignment (DFA) trains hidden layers with fixed random projections of the output error, avoiding the transposed-weight backward pass of backpropagation (BP). We study a failure mode of DFA training that is distinct from feedback quality: the local weight update is calculated by an outer product, so anisotropy can enter through either its presynaptic-activity factor or its local-error factor. Our analyses with controlled synthetic regimes isolate the first failure mode and show an approximately 40-percentage-point activity-conditioning gain when high-variance directions contain task-irrelevant nuisance. Three clean confirmations isolate a different regime: error conditioning improves raw DFA by 1.77--7.53 percentage points, and combining independently selected activity and error factors adds 0.40--0.90 points over activity conditioning. The signs hold for tanh/one-vs-rest MNIST and preregistered Fashion-MNIST, and replicate on eight fresh seeds in a ReLU/softmax MNIST model. This factorization yields a symmetric block-local family of normalized DFA (nDFA): activity nDFA right-preconditions by an inverse activity second moment, error nDFA left-preconditions by an inverse local-error second moment, and K-nDFA applies both factors with separately tuned damping. A linearized post-alignment calculation gives an exact input-side spectral identity and a Kronecker-factor motivation for the two-sided rule, whereas norm matching rules out a scalar step-size explanation. The error factor is fragile when under-damped, BatchNorm is a strong activity-side alternative, and convnet gains remain partial. We therefore frame conditioned DFA as a factor-level study of when local outer-product rules fail, not as a general replacement for BP or a solution to all-layer convolutional credit assignment.

Houman Safaai, Varun Reddy, Bernardo L. Sabatini

Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University · Department of Neurobiology, Howard Hughes Medical Institute, Harvard Medical School

Multilayer Perceptrons Neural Network Training Dynamics

Ask PDF

Jun 9, 2026

Overcoming Rank Collapse in Feedback Alignment

Backpropagation (BP) is widely viewed as biologically implausible, in part because it requires feedback weights to be the transpose of forward weights for error propagation. Interestingly, when training a network with fixed random feedback weights to circumvent this issue, learning aligns the forward weights with the feedback weights, leading the backpropagated error signal to become an approximation of the standard gradient used by BP. This process, called Feedback Alignment (FA), occurs in MLPs and very shallow CNNs but does not scale well to deeper architectures. In this work, we first investigated differences between BP and FA models, trained on CIFAR10, specifically focusing on the effective rank of the signal. We found that the FA error has a considerably lower rank and hence is constrained to a lower-dimensional subspace compared to BP, limiting exploration of the parameter space. Motivated by this observation, we evaluated two mechanisms for increasing the effective dimensionality of FA: Muon, an optimiser that orthogonalises weight updates; and hidden activity normalisation, which promotes activation orthogonality. Across larger architectures and benchmarks, we find that these methods consistently improve over FA baselines, for example, on CIFAR100 with a Resnet-18, accuracy increases by 9 percentage points. Our results identify low-dimensional gradient dynamics as a key obstacle to scaling FA and suggest that inducing higher-dimensional update geometry is a promising route toward scaling alternatives to backpropagation.

Gauthier Boeshertz, Razvan Pascanu, Claudia Clopath

Department of Bioengineering Imperial College London · Mila

Backpropagation Neural Network Optimization Neural Network Training Dynamics

Ask PDF

Oct 27, 2026

Nonlinear Direct Feedback Alignment for Scalable Backpropagation-Free Training

Backpropagation (BP) remains the predominant algorithm for training deep neural networks, but it is biologically implausible and suffers from the backward locking problem, constraining parallelization across layers. Direct Feedback Alignment~(DFA), which propagates the global output error directly to each hidden layer via fixed feedback matrices, has emerged as a promising alternative for addressing these limitations. However, a significant performance gap persists between DFA and BP, which we attribute to the limited ability of fixed feedback matrices to capture the nonlinear transformation from hidden representations to the network output. To address this limitation, we propose Nonlinear Direct Feedback Alignment (NDFA), which constructs each hidden-layer feedback using a lightweight nonlinear surrogate subnetwork. To enhance the effectiveness of the feedback throughout training, we further propose a learning strategy to update the surrogate subnetwork, thereby better approximating the underlying nonlinear transformation. Extensive experiments across residual networks and vision transformers on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets demonstrate that NDFA consistently outperforms existing DFA-based methods and substantially narrows the performance gap to BP, while maintaining efficient parallel training. The code is available at https://github.com/YanFan321/NDFA.

Wu Lin, Yan Fan, Chenxiang Ma +4

College of Computer Science and Software Engineering, Shenzhen University · School of Artificial Intelligence, Tianjin University · School of Artificial Intelligence, Shenzhen University +2

Neural Network Optimization

Ask PDF

Abstract

Neural Network Training Dynamics

Explore similar work

Conditioned Direct Feedback Alignment via Activity and Error Geometry

Overcoming Rank Collapse in Feedback Alignment

Nonlinear Direct Feedback Alignment for Scalable Backpropagation-Free Training