Selectivity and Shape in the Design of Forward-Forward Goodness Functions

Talha Ruzgar Akkus; Suayp Talha Kocabay; Kamer Ali Yuksel; Hassan Sawaf

cs.LGMar 28, 2026

Selectivity and Shape in the Design of Forward-Forward Goodness Functions

Authors: Talha Ruzgar Akkus, Suayp Talha Kocabay, Kamer Ali Yuksel, Hassan Sawaf

Organizations: aiXplain, Inc., San Jose, CA

Abstract

The Forward-Forward (FF) algorithm trains networks layer-by-layer using a local "goodness function," yet sum-of-squares (SoS) has remained the only choice studied. We systematically explore the goodness-function design space and identify a unifying principle: the goodness function must be sensitive to the shape of neural activity, not its total energy. This principle is motivated by the observation that deep network activations follow heavy-tailed distributions and that discriminative information is often concentrated in peak activities. We propose two complementary families: selective functions (top-k, entmax-weighted energy) that measure only peak activity, and shape-sensitive functions (excess kurtosis / "burstiness" and higher-order moments) that reward heavy-tailed distributions via scale-invariant statistics. Combined with separate label-feature forwarding (FFCL), controlled experiments across 13 goodness functions, 5 activations, 6 datasets, and three continuous sweeps, each tracing a characteristic inverted-U, yield 89.0% on Fashion-MNIST and 98.2+-0.1% on MNIST (4x2000), a +32.6pp gain over SoS, with consistent improvements across all benchmarks (+72pp USPS, +52pp SVHN). The scale-invariant nature of burstiness makes it particularly robust to magnitude shifts across layers and datasets.

Figures & tables

Label	Goodness function	Acc%	$\Delta$
Std	SoS (ReLU) ( Hinton, 2022 )	56.41	—
Std	SoS (GELU)	61.43	+5.0
Std	Softmax-energy-margin ( Shah and Tripathi, 2025 )	68.72	+12.3
Std	Contrast top- $k$ (GELU)	70.49	+14.1
Std	Top- $k$ (Swish)	79.03	+22.6
Std	LayerNorm-top- $k$ (GELU)	83.28	+26.9

Table 1: Test accuracy (%) on Fashion-MNIST (4

\times

2000). Best activation/norm-gate per row.

\Delta

: improvement over ReLU+SoS baseline.

Goodness function	Standard	FFCL
SoS ( Hinton, 2022 )	61.43	82.38
Variance	61.55	81.74
Game-theoretic †	61.37	82.38
Neg. entropy	67.39	80.43
Softmax-energy-margin †	69.85	81.89
Contrast top- $k$	70.49	83.59

Table 2: Best test accuracy (%) per goodness function on Fashion-MNIST (4

\times

2000). † From Shah and Tripathi (2025) . ‡ GELU,

\alpha{=}1.5

. § Parameter-free.

Panel of Figure 1 — Figure 1: Three sweep axes on Fashion-MNIST (4 $\times$ 2000). All three trace an inverted-U. (a) $k$ -sweep: FFCL robust ( $<$ 2pp). (b) $\alpha$ -sweep: peaks at $\alpha{\approx}1.5$ . (c) Moment- $p$ : peaks at $p{\approx}5$ – $6$ (89.04% FFCL).

	SoS	Top- $k$	Entmax-1.5	Burstiness
ReLU	56.41	—	—	—
GELU	61.43	77.65	85.08	87.15
Swish	55.99	79.03	—	88.07
LN-GELU	61.49	84.00	85.41	87.66
LN-Swish	65.06	84.35	85.42	87.98

Table 3: Goodness

\times

activation interaction (Fashion-MNIST, 4

\times

2000, standard FF).

	MNIST	F-MNIST	CIFAR-10	USPS	SVHN	EMNIST
SoS Baseline (ReLU)	88.76	56.41	37.85	21.87	32.25	69.91
FFCL + SoS (GELU)	93.45	82.38	46.46	76.83	71.65	77.80
Std + Top- $k$ (Swish)	90.23	78.64	37.37	82.86	57.68	66.35
FFCL + Ctrst. top- $k$	93.31	83.59	47.07	90.03	66.05	79.69
Std + Burstiness (Swish)	96.54	88.06	53.56	93.12	78.85	86.21
FFCL + Burstiness (GELU)	98.09	88.41	54.36	93.27	82.84	91.62

Table 4: Test accuracy (%) across six datasets (4

\times

2000, seed 42).

\Delta

: improvement over SoS baseline.

	MNIST	F-MNIST	EMNIST	USPS
SoS Baseline	89.17 $\pm$ 0.30	58.67 $\pm$ 2.19	69.97 $\pm$ 0.34	22.21 $\pm$ 0.96
FFCL + Burstiness	98.18 $\pm$ 0.08	88.36 $\pm$ 0.23	91.50 $\pm$ 0.16	93.52 $\pm$ 0.22
$\Delta$ (mean)	+9.01	+29.69	+21.53	+71.31

Table 5: Seed sensitivity: mean

\pm

std over 5 seeds (4

\times

2000).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

#	Label	Act.	Goodness	Acc%
1	FFCL	GELU	burstiness	98.09
2	FFCL	GELU	LN-burstiness	98.09
3	FFCL	Swish	burstiness	98.00
4	FFCL	Swish	LN-burstiness	98.00
5	Std	Swish	burstiness	96.54
6	Std	Swish	LN-burstiness	96.45

Appendix

Table 6: Test accuracy (%) on MNIST (4

\times

2000), ranked by accuracy. Best of norm-gate on/off reported for each configuration. Highlighted : burstiness variants.

#	Label	Act.	Goodness	Acc%
1	FFCL	GELU	burstiness	88.41
2	FFCL	GELU	LN-burstiness	88.41
3	FFCL	Swish	burstiness	88.21
4	FFCL	Swish	LN-burstiness	88.21
5	Std	Swish	LN-burstiness	88.11
6	Std	Swish	burstiness	88.07

Appendix

Table 7: Complete ranked results on Fashion-MNIST (4

\times

2000). Best of norm-gate on/off reported. Highlighted : burstiness variants.

Label	Act.	Goodness	Acc%
FFCL	LN-GELU	LN-burstiness	88.71
FFCL	LN-GELU	burstiness	88.69
FFCL	LN-Swish	burstiness	88.68
FFCL	LN-Swish	LN-burstiness	88.66
FFCL	GELU	burstiness	88.41
FFCL	GELU	LN-burstiness	88.41

Appendix

Table 9: Burstiness variants on Fashion-MNIST (4

\times

2000, best of norm-gate on/off). Burstiness and LN-burstiness produce nearly identical results, confirming the scale-invariance of excess kurtosis. LN-GELU and LN-Swish activations improve over GELU/Swish with FFCL.

Goodness	2 $\times$ 500	4 $\times$ 2000	$\Delta$	2 $\times$ 500	4 $\times$ 2000	$\Delta$
	MNIST			Fashion-MNIST
SoS (ReLU)	89.62	88.76	$-$ 0.86	61.07	56.41	$-$ 4.66
Contrast top- $k$	87.97	86.90	$-$ 1.07	69.04	70.49	+1.45
Top- $k$	89.63	90.37	+0.74	76.65	79.03	+2.38

Appendix

Table 11: Architecture scaling: accuracy change from 2

\times

500 to 4

\times

2000 (standard FF, best activation per goodness).

Goodness	Standard	FFCL	$\Delta$
SoS	61.43	82.38	+20.95
Game-theoretic	61.37	82.38	+21.01
Variance	61.55	81.74	+20.19
Entropy	67.39	80.43	+13.04
Softmax-energy-margin	69.85	81.89	+12.04
Contrast top- $k$	70.49	83.59	+13.10

Appendix

Table 12: FFCL lift per goodness function on Fashion-MNIST (4

\times

2000).

Forward-Forward Learning

Forward-Forward learning trains neural networks with layer-local goodness objectives using forward passes instead of backpropagated gradients.

See all 17 papers

What Does Goodness Measure? A Likelihood-Ratio Account of Forward-Forward Learning
Jul 14, 2026Paolo GiannitrapaniMahalanobis Distance Likelihood Ratio Testing
Department of Information Engineering, Electronics and Telecommunications (DIET) Sapienza University of Rome Rome, Italy
Covariance-Aware Goodness for Scalable Forward-Forward Learning
May 5, 2026Xiaoyi Jiang, Bashir M. Al-Hashimi, Kai XuConvolutional Neural Networks Local Learning
King’s College London, United Kingdom · University of Nottingham
Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant
May 7, 2026Amirhossein YousefiramandiNeural Network Training Dynamics Local Learning

Jul 14, 2026J/K move · Enter open · S save

What Does Goodness Measure? A Likelihood-Ratio Account of Forward-Forward Learning

Paolo Giannitrapani

Department of Information Engineering, Electronics and Telecommunications (DIET) Sapienza University of Rome Rome, Italy

Mahalanobis Distance Likelihood Ratio Testing Forward-Forward Learning

Open paper AskPDF arXiv

The Forward-Forward (FF) algorithm trains each layer locally, so that a scalar goodness - the sum of squared activations - is high on real inputs and low on contrastive ones, with activations normalized between layers. Both choices are usually treated as heuristics. Under an explicit generative model they are not: the squared goodness is the sufficient statistic of a likelihood-ratio test between two zero-mean populations differing in scale, and the FF threshold is its boundary. It generalizes: anisotropic populations yield a Mahalanobis goodness, the plain square being its isotropic case; heavy-tailed populations yield a saturating statistic whose slope is a posterior precision - divisive normalization - with bounded evidence and an advantage only under aggregation. The same lens characterizes the inter-layer normalization: it must remove the length while preserving per-coordinate energy, explaining a depth collapse we observe under unit-norm normalization; and the pairwise objective admits a scale-inflation shortcut that a whitened goodness removes.

Abstract

Forward-Forward Learning

Explore similar work

What Does Goodness Measure? A Likelihood-Ratio Account of Forward-Forward Learning

Covariance-Aware Goodness for Scalable Forward-Forward Learning

Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant