The Price of Locality: Why Forward-Forward Underperforms Backpropagation?

Zhaoxian Wu; Haichuan Liu; Tianyi Chen

cs.LGSep 27, 2026

The Price of Locality: Why Forward-Forward Underperforms Backpropagation?

Authors: Zhaoxian Wu, Haichuan Liu, Tianyi Chen

Organizations: Cornell Tech and Cornell University

Abstract

The Forward-Forward Algorithm (FFA) replaces backpropagation (BP) with layer-wise local contrastive objectives, eliminating the backward pass and the need to retain intermediate activations, yet suffers a persistent performance gap with BP that worsens with depth. This paper diagnoses two structural failure modes: an optimization floor arising from concurrent local updates; and a geometric collapse of layer representations driven by the local update mechanism. On the optimization side, we prove that the FFA loss satisfies the Polyak--Lojasiewicz inequality at each layer; however, simultaneous layer updates induce inter-layer representation-distribution drift, so each layer optimizes against a moving input distribution and incurs an error floor. On the representational side, the pairwise similarity kernel of layer representations contracts exponentially toward rank one as depth increases, collapsing the diversity of per-layer error signals. This collapse bounds FFA's effective learning capacity, which measures the diversity of gradient information across layers, independently of depth, whereas BP's chain-rule signal preserves per-layer diversity, yielding a capacity that scales with depth.

Figures & tables

Panel of Figure 1 — Figure 1: Left: Multi-layer FFA convergence in the toy model. The vertical dotted line marks $k=10^{3}$ . Right: Test accuracy curve of ResNet-18 @ CIFAR-10 training.

Figure 2: CIFAR-10 classification accuracy across ResNet depths (ResNet18/24/56), measured by a detached readout. BP remains best at every depth; supervised local-learning methods trail by a moderate margin, while goodness-based FFA methods trail substantially.

Blocks $\mathbin{\times}$ layers/block	Best acc.	Final loss	Mean $\erank(\Gamma)$
$12\mathbin{\times}1$	$53.02\pm 0.48\%$	$1.1014\pm 0.0034$	$2.198\pm 0.042$
$6\mathbin{\times}2$	$58.09\pm 1.85\%$	$1.0455\pm 0.0068$	$5.146\pm 0.592$
$4\mathbin{\times}3$	$67.57\pm 4.35\%$	$1.0064\pm 0.0158$	$4.856\pm 0.552$
$3\mathbin{\times}4$	$72.22\pm 1.07\%$	$0.9795\pm 0.0018$	$5.135\pm 2.164$
$2\mathbin{\times}6$	$76.21\pm 0.48\%$	$0.9657\pm 0.0027$	$7.613\pm 3.828$
$1\mathbin{\times}12$	$78.74\pm 4.06\%$	$0.9550\pm 0.0199$	$7.975\pm 1.214$

Table 1: Controlled locality intervention on CIFAR-10 with CNN12. Entries are mean

\pm

sample standard deviation over three seeds.

Scale	Params	Budget	BP	FFA	LCE	FFA/BP	LCE/BP
tiny ( $L{=}2$ )	7M	0.14B	138	516	166	3.73 $\times$	1.20 $\times$
small ( $L{=}4$ )	16M	0.32B	68	313	84	4.59 $\times$	1.22 $\times$
medium ( $L{=}6$ )	30M	0.60B	48	275	64	5.74 $\times$	1.33 $\times$
large ( $L{=}8$ )	51M	1.02B	39	256	54	6.55 $\times$	1.38 $\times$
xlarge ( $L{=}12$ )	124M	2.48B	28	258	42	9.19 $\times$	1.51 $\times$

Table 2: Perplexity of Transformer pre-training on OpenWebText under Chinchilla-style budgets.

Table 3: (Left) Controlled spectral interpolation on CNN3/CIFAR-10. Entries report best test accuracy and terminal mean layerwise participation-ratio

\erank(\Gamma)

. (Right) Finite-sample Jacobian-concentration diagnostics of the residual model ( 1 ). Each entry reports mean

\pm

sample standard deviation over three seeds.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Training-time last-block rank diagnostics on CNN@CIFAR-10. Top: representation effective rank

\erank(\Sigma)

. Bottom: error-signal effective rank

\erank(\Gamma)

. The

\Sigma

probe contains one row-normalized representation per image.

Architecture	BP	Vanilla FFA	Gap
CNN (3 conv layers)	76.03%	73.73%	2.30%
ViT (2 transformer layers)	72.71%	64.77%	7.94%

Appendix

Table 4: Test accuracy on modern architectures.

Arch	Conv blocks	Channel growth	Spatial scales	Params (BP)
CNN3	3	$3{\to}32{\to}64{\to}128$	$32^{2}{\to}16^{2}{\to}8^{2}$	0.62M
CNN6	6	$3{\to}32{\to}32{\to}64{\to}64{\to}128{\to}128$	$32^{2}{\to}16^{2}{\to}8^{2}$	0.81M
CNN9	9	$32{\to}64{\to}128{\to}256$ (VGG-style)	$32^{2}{\to}16^{2}{\to}8^{2}{\to}4^{2}$	2.81M

Appendix

Table 5: CNN backbone architectures used in the unified benchmark.

Algorithm	Category	CNN3	CNN6	CNN9
Standard BP	Baseline	79.26 $\pm$ 0.24	85.30 $\pm$ 0.15	86.12 $\pm$ 0.28
Vanilla FFA [ 1 ]	FFA	47.98 $\pm$ 1.71	52.31 $\pm$ 1.29	54.05 $\pm$ 1.99
SymBa [ 14 ]	FFA-contrastive	60.96 $\pm$ 1.65	61.93 $\pm$ 0.90	63.69 $\pm$ 0.50
SCFF [ 16 ]	FFA-contrastive	50.04 $\pm$ 1.95	50.75 $\pm$ 1.99	53.47 $\pm$ 1.85
Trifecta [ 47 ]	FFA-extended	57.51 $\pm$ 0.57	60.52 $\pm$ 0.34	62.42 $\pm$ 1.05
Scodellaro CNN-FFA [ 8 ]	FFA-extended	38.41 $\pm$ 0.65	43.12 $\pm$ 0.97	43.58 $\pm$ 2.38

Appendix

Table 6: CIFAR-10 CNN benchmark across local learning algorithms. Entries are best test accuracy (%, mean

\pm

std over seeds 0/1/2).

Dataset	Depth	BP	LCE	SFF	DF	$\Delta_{\text{BP}-\text{best}}$
MNIST	R18	99.66	99.62	99.60	99.64	$+0.02$
	R24	99.71	99.65	99.69	99.68	$+0.02$
	R56	99.67	99.70	99.70	99.67	$-0.03$
	R108	99.68	99.66	99.70	99.64	$-0.02$
CIFAR-10	R18	95.15	89.06	87.20	87.16	$+6.09$
	R24	95.42	89.03	87.09	87.19	$+6.39$

Appendix

Table 7: Detached-readout accuracy (%) of BP-competitive methods on ResNet-{18,24,56,108} across four task-difficulty regimes. “base.”/“hard.” denote the baseline / hardened training recipe.

\Delta

is the gap from BP to the best local method on the row. Single seed throughout.

Figure 4: Detached-readout best test accuracy versus ResNet depth across four task-difficulty regimes (MNIST and CIFAR-10 use the baseline recipe; CIFAR-100 and Tiny-ImageNet use the hardened recipe). The BP–local-learning gap is essentially zero on MNIST (all four BP-competitive methods saturate), $5$ – $6$ pp on CIFAR-10 for BP-competitive locals (and ${>}50$ pp for the label-injection FFA family; see Table 9 ), and grows to $8$ – $13$ pp on CIFAR-100/Tiny-ImageNet—where added depth and the hardened recipe partially reshape, but do not close, the gap.

Figure 5: Headline view of the data in Table 7 : BP minus best BP-competitive-local detached-readout accuracy (pp) plotted across the six (dataset, recipe) regimes ordered by approximate difficulty. One line per ResNet depth. The gap is essentially zero on MNIST (BP saturates), $5$ – $6$ pp on CIFAR-10, and rises into the $7$ – $15$ pp band on CIFAR-100 / Tiny-ImageNet across both recipes. At every regime, the within-depth spread is $\leq 3$ pp—small relative to the difficulty-driven trend. Single-seed estimates throughout.

Dataset	Depth	native	read.	$\Delta_{r-n}$	native	read.	$\Delta_{r-n}$	native	read.	$\Delta_{r-n}$	native	read.	$\Delta_{r-n}$
		BP			LCE			SFF			DF
C-100 (hard.)	R18	79.85	79.79	$-0.06$	67.98	68.48	$+0.50$	69.61	68.00	$-1.61$	67.31	67.84	$+0.53$
	R24	80.38	80.40	$+0.02$	68.80	68.92	$+0.12$	70.07	68.19	$-1.88$	69.03	69.47	$+0.44$
	R56	79.77	79.74	$-0.03$	70.16	70.89	$+0.73$	71.88	69.84	$-2.04$	71.25	72.00	$+0.75$
	R108	79.91	80.19	$+0.28$	71.02	71.48	$+0.46$	72.81	70.59	$-2.22$	72.56	72.43	$-0.13$
T-IN (hard.)	R18	65.06	64.69	$-0.37$	50.55	51.26	$+0.71$	52.67	51.77	$-0.90$	51.08	51.52	$+0.44$

Appendix

Table 8: Native test accuracy versus detached-readout accuracy (%) on CIFAR-100 and Tiny-ImageNet under the hardened recipe. “

\Delta_{r-n}

” is readout minus native; positive means the detached readout extracts more class information from the same trunk than the algorithm’s native head does. The pattern is algorithm-specific: BP and DF show

|\Delta_{r-n}|<1

pp; LCE’s readout exceeds its native head by up to

+1.0

pp; SFF’s native head consistently exceeds its readout by up to

-3.20

pp because SFF’s native logits aggregate all per-block heads.

Dataset	Depth	SCFF	Vanilla FFA	SymBa	LayerCollab	Trifecta
MNIST	R18	79.44	75.32	91.43	84.76	90.87
	R24	82.87	84.16	91.23	82.40	89.88
	R56	84.83	81.45	87.73	72.47	90.12
	R108	67.60	81.47	87.25	89.94	90.98
CIFAR-10	R18	32.09	31.69	41.46	32.45	41.41
	R24	33.66	31.14	40.80	32.45	39.52

Appendix

Table 9: Label-injection FFA family detached-readout accuracy (%) on the same ResNet ladder. These methods are excluded from CIFAR-100 / Tiny-ImageNet, where they failed to train stably across depths under their published recipes.

Dataset	Depth	BP	LCE	SFF	DF	avg
CIFAR-100	R18	$+2.05$	$+3.84$	$+5.02$	$+5.28$	$+4.05$
	R24	$+2.38$	$+4.54$	$+5.07$	$+6.77$	$+4.69$
	R56	$+1.83$	$+5.62$	$+6.29$	$+8.04$	$+5.45$
	R108	$+2.00$	$+4.37$	$+6.65$	$+7.16$	$+5.05$
Tiny-ImageNet	R18	$+8.64$	$+6.77$	$+7.21$	$+6.15$	$+7.19$
	R24	$+8.46$	$+6.40$	$+7.86$	$+6.67$	$+7.35$

Appendix

Table 10: Detached-readout accuracy gain (pp) from baseline to hardened recipe, per (depth, algorithm). Last column averages across the four BP-competitive algorithms.

Forward-Forward Learning

Forward-Forward learning trains neural networks with layer-local goodness objectives using forward passes instead of backpropagated gradients.

See all 16 papers

HCL-FF: Hierarchical and Contrastive Learning for Forward-Forward Algorithm
May 24, 2026Jie-En Yao, Hong-En Chen, C. -C. Jay KuoHierarchical Representation Learning Forward-Forward Learning
University of Southern California
Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training
Jun 4, 2026Yucheng ChenBackpropagation Local Learning
Amplimit Toronto, ON, Canada
Covariance-Aware Goodness for Scalable Forward-Forward Learning
May 5, 2026Xiaoyi Jiang, Bashir M. Al-Hashimi, Kai XuConvolutional Neural Networks Local Learning
King’s College London, United Kingdom · University of Nottingham

May 24, 2026J/K move · Enter open · S save

HCL-FF: Hierarchical and Contrastive Learning for Forward-Forward Algorithm

Jie-En Yao, Hong-En Chen, C. -C. Jay Kuo

University of Southern California

Hierarchical Representation Learning Forward-Forward Learning Image Classification

Open paper AskPDF arXiv

Deep neural networks trained with backpropagation have achieved outstanding performance in vision tasks but remain biologically implausible, computationally demanding, and difficult to interpret. The Forward-Forward (FF) algorithm offers a promising alternative by training each layer independently through local goodness objectives. However, its purely local optimization lacks hierarchical coordination across layers, and the decoupling of goodness from features leaves the representations unconstrained and semantically ambiguous. We propose a Hierarchical and Contrastive Learning FF framework (HCL-FF) to address these limitations. HCL-FF introduces (1) a coarse-to-fine hierarchical learning strategy that guides representations from low-level cues to high-level semantics, and (2) a supervised contrastive objective that enforces class-discriminative alignment after goodness decoupling. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that HCL-FF achieves new state-of-the-art performance among FF-based methods, with notable accuracy gains of +5.46%, +17.00%, and +12.51%, respectively.

Abstract

Forward-Forward Learning

Explore similar work

HCL-FF: Hierarchical and Contrastive Learning for Forward-Forward Algorithm

Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training

Covariance-Aware Goodness for Scalable Forward-Forward Learning