Rescuing Points Trapped Inside Incorrect Decision Regions: A target for Evaluating and Improving Forward-Forward Learning

Alireza Rahmansetayesh; Amirhossein Yousefiramandi; Maxwell Spiridonov; Sumit Raut; Kent Ziti; Pierre Baldi

cs.LGOct 12, 2026

Rescuing Points Trapped Inside Incorrect Decision Regions: A target for Evaluating and Improving Forward-Forward Learning

Authors: Alireza Rahmansetayesh, Amirhossein Yousefiramandi, Maxwell Spiridonov, Sumit Raut, Kent Ziti, Pierre Baldi

Organizations: Department of Computer Science, University of California, Irvine, Irvine, CA, USA · Independent Researcher

Abstract

Forward-Forward learning replaces end-to-end backpropagation with local, layer-wise training objectives. Its variants are evaluated primarily by classification accuracy, often measured with linear probes trained on their representations. We show that this evaluation can over-credit the learning method: a linear probe on the representations of an untrained convolutional neural network already reaches accuracy well above chance. Moreover, a single accuracy value does not reveal which datapoints become correctly classified after training and which remain misclassified. To examine what learning contributes beyond the random, untrained baseline, we present an analysis methodology built on the Boundary Proximity Index (BPI), a multiclass margin normalized by a class-centroid reference. We apply our analysis to seven Forward-Forward variants and related layer-local methods and find that initially misclassified points near the probe's decision boundary generally have higher rescue rates than those farther inside incorrect decision regions. Rescue rates vary substantially across methods, with some correcting a substantial fraction of these deeply embedded points. Together, these analyses offer a way to evaluate representation learning beyond accuracy and identify the rescue of points far inside incorrect decision regions as a target for improving local learning methods.

Figures & tables

Figure 1: Layer-wise ablation of training. Each panel is one method and dataset (mean and standard deviation over three seeds). A curve is the test accuracy of a linear probe against the depth at which it reads features, for one training condition: the color encodes how many of the first layers are trained, from none (yellow, the untrained network) to all (dark purple, the usual reported result), the rest keeping their random initialization. The gap between the yellow and the dark-purple curve is the improvement from training under this probe protocol. The SCFF networks include two appended layers (Appendix ). The ImageNet-1K panel (HFF-VGG16, trained by us) shows five of the 17 conditions (0, 4, 8, 12 and 16 layers trained).

Figure 2: Rescue and breakage rates against boundary proximity on CIFAR-10 (test set), one curve per method. Each point is the mean rescue rate (a, b) or breakage rate (c, d) over the datapoints of one BPI bin (30 equal-width bins on

[-2,0)

for rescue and on

[0,2)

for breakage, plotted at the bin center; the outermost bin also collects datapoints beyond

-2

2

). Rows are the first four layers (a, c) or layer transitions (b, d); the first transition,

0{\to}1

, compares a linear probe on the input pixels with the first layer. Shaded bands are nominal 95% Wilson intervals from pooled example-level counts, a descriptive binwise uncertainty that ignores dependencies across seeds and networks (Appendix ); bins with fewer than 20 datapoints are omitted. (a) Across training: fraction of datapoints misclassified by the untrained network’s probe at layer

\ell

that are correct after training, against BPI in the untrained network. (b) Across depth: fraction misclassified at layer

\ell

of the trained network that are correct at layer

\ell{+}1

, against BPI at layer

\ell

. (c,d) Breakage for the same comparisons (BPI is positive for correctly classified points). BP: backpropagation on the SCFF architecture, or VGG16 trained from scratch (three seeds). For VGG16 (pretrained), “training” is ImageNet pretraining (one set of weights, so no band). Every layer is shown in , and the same figure for CIFAR-100, Tiny ImageNet and ImageNet-1K in Appendix .

Figure 3: Rescue (a) and breakage (b) from the first to the deepest layer of the trained network, on CIFAR-10 (test set), against BPI at the first layer. In (a) the datapoints are misclassified by the first-layer probe and the rate is the fraction correct at the method’s deepest layer (depths in ); in (b) they are correct at the first layer and the rate is the fraction misclassified at the deepest layer. Curves, bands and omitted bins as in . CIFAR-100 and Tiny ImageNet: and .

Figure 4: Null controls for VGG16 on CIFAR-10, in the format of (first four layers; every layer is shown in ). Trained: VGG16 from scratch on CIFAR-10 and pretrained VGG16; the other curves come from the untrained VGG16 (three initializations). Null 1: two probes with different seeds on the same untrained features (none for row

0{\to}1

). Null 2 (a,c): untrained network

A

against

B

(all ordered pairs). Null 3 (b,d): layers

\ell

and

\ell{+}1

of one untrained network (row

0{\to}1

: pixels and layer 1). The controls for every other method are in Appendix , and those for ImageNet-1K in .

Figure 5: Null controls for the first-to-last comparison of , for VGG16 on CIFAR-10. The untrained curve uses the first and the deepest layer of the same untrained network (null 3). Colors as in . Other methods: Appendix .

Figure 6: Where in depth training builds linear separability, on CIFAR-10 (test set; mean over seeds, bands are the minimum and maximum over seeds). Total rescue minus total breakage, in % of the test set, which equals the change in probe accuracy. (a) Across training, at layer

\ell

: the probe accuracy of the trained network minus that of the untrained network of the same seed. (b) Across depth, at layer

\ell

: the probe accuracy of the trained network at depth

\ell

minus that at depth

\ell{-}1

(for

\ell=1

, minus the raw-pixel probe), i.e. the layer added to the earlier ones. BP (ASGE arch.): backpropagation on ASGE’s ten-block architecture. Other curves as in . CIFAR-100 and Tiny ImageNet: and .

Figure A.1: As , with a row for every layer.

Figure A.2: As , on CIFAR-100, with a row for every layer.

Figure A.3: As , on Tiny ImageNet, with a row for every layer.

Figure A.4: As , on ImageNet-1K, with a row for every layer: HFF-VGG16 (our training), pretrained VGG16 and untrained VGG16 (nulls 1 to 3 of ; Appendix ).

Figure A.5: As , on CIFAR-100.

Figure A.6: As , on Tiny ImageNet.

Figure A.7: As , on CIFAR-100 (BP on the SCFF architecture is available only across depth).

Figure A.8: As , on Tiny ImageNet (BP on the SCFF architecture is available only across depth).

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Local objective	Layers	Implementation
SCFF ( Chen et al., 2025 )	self-contrastive goodness	3	official code
Trifecta ( Dooms et al., 2024 )	symmetric goodness loss, batch norm (no overlapping updates)	6	re-implemented
DeeperForward ( Sun et al., 2025 )	layer-normalized mean goodness	4	official code
HCL-FF ( Yao et al., 2026 )	hierarchical goodness + supervised contrastive	4	official code
HFF ( Sarode et al., 2026 )	class prototypes on the hypersphere	3	re-implemented
Mono-Forward ( Gong et al., 2025a )	layer-wise cross-entropy, detached	4	re-implemented

Appendix

Table A.1: Methods analyzed. “Layers” is the number of probed layers on CIFAR-10 (SCFF, BP and Mono-Forward use five layers on Tiny ImageNet); for DeeperForward and HCL-FF a layer is one residual block. BP is backpropagation on the SCFF architecture with a linear classification layer; the pretrained VGG16 is compared with its random initialization and the from-scratch VGG16 with the initialization it was trained from.

Figure A.9: Null controls for VGG16 on CIFAR-10: as with every layer.

	Across training, layer 3		Across depth, $1{\to}2$		First $\to$ last
Method	$\mathrm{BPI}<-1$	within 0.5	$\mathrm{BPI}<-1$	within 0.5	all
SCFF	7.1/0.9/0.0	50.7/33.6/4.6	0.7/0.7/0.0	44.4/37.2/2.8	43.9/34.6
Trifecta	2.3/0.0/0.0	38.3/20.9/4.6	7.7/0.2/0.0	50.1/24.2/3.0	58.2/34.6
DeeperForward	44.3/5.7/0.0	74.9/36.1/3.2	1.1/1.5/0.0	48.5/37.3/2.5	50.2/32.6
HCL-FF	59.8/5.7/0.0	84.5/35.7/3.3	5.3/1.1/0.0	50.5/37.1/2.5	52.7/34.9
HFF	22.1/1.9/0.0	47.5/32.0/4.4	7.2/0.9/0.0	46.5/34.4/3.0	37.3/31.7

Appendix

Table A.2: Rescue rate (%) on CIFAR-10 with seeds pooled, as “trained / null 2 / null 1” for the across-training column and “trained / null 3 / null 1” for the across-depth column. Across training, layer 3 : datapoints misclassified by the layer-3 probe of the untrained network, against BPI in that network. Across depth,

1{\to}2

: datapoints misclassified at layer 1, rescued at layer 2 (null 1: layer 1 with a second probe). First

\to

last (trained / null 3): all datapoints misclassified at the first layer that are correct at the deepest layer. Columns “

\mathrm{BPI}<-1

” and “within 0.5” restrict to datapoints with BPI below

-1

or in about

[-0.5,0)

; they are computed on the 30 BPI bins of width

1/15

used for the curves (bins with midpoint above

-0.5

), which differs from exact thresholds by less than 1 point.

Figure A.10: Null controls for SCFF on CIFAR-10: as with every layer (top) and as (bottom).

Figure A.11: Null controls for BP (SCFF arch.) on CIFAR-10: as with every layer (top) and as (bottom).

Figure A.12: Null controls for Trifecta on CIFAR-10: as with every layer (top) and as (bottom).

Figure A.13: Null controls for ASGE on CIFAR-10: as with every layer (top) and as (bottom).

Figure A.14: Null controls for DeeperForward on CIFAR-10: as with every layer (top) and as (bottom).

Figure A.15: Null controls for HFF on CIFAR-10: as with every layer (top) and as (bottom).

Figure A.16: Null controls for HCL-FF on CIFAR-10: as with every layer (top) and as (bottom).

Figure A.17: Null controls for Mono-Forward on CIFAR-10: as with every layer (top) and as (bottom).

CIFAR-100
	Across training, layer 3		Across depth, $1{\to}2$		First $\to$ last
Method	$\mathrm{BPI}<-1$	within 0.5	$\mathrm{BPI}<-1$	within 0.5	all
BP (VGG16, scratch)	10.4/0.1/0.0	48.7/6.0/0.8	0.6/0.1/0.0	32.2/10.2/0.8	58.9/27.2
VGG16 (pretrained)	7.8/0.1/0.0	43.6/6.0/0.8	0.1/0.1/0.0	24.1/10.2/0.8	65.7/27.2
SCFF	2.3/0.2/0.0	31.0/20.0/3.7	0.3/0.3/0.0	29.8/23.4/2.4	26.9/19.3
Trifecta	0.4/0.0/0.0	21.8/10.7/1.5	2.5/0.0/0.0	33.0/16.5/1.2	31.7/21.2

Appendix

Table A.3: Rescue rate (%) of the null controls on CIFAR-100 and Tiny ImageNet, in the format of : “trained / null 2 / null 1” for the across-training column and “trained / null 3 / null 1” for the across-depth column, seeds pooled; First

\to

last : trained / null 3. For BP (VGG16, scratch) and VGG16 (pretrained) the nulls are those of the untrained VGG16 of the pretrained pipeline.

Figure A.18: Null controls for VGG16 on CIFAR-100: as with every layer (top) and as (bottom).

Figure A.19: Null controls for SCFF on CIFAR-100: as with every layer (top) and as (bottom).

Figure A.20: Null controls for Trifecta on CIFAR-100: as with every layer (top) and as (bottom).

Figure A.21: Null controls for ASGE on CIFAR-100: as with every layer (top) and as (bottom).

Figure A.22: Null controls for DeeperForward on CIFAR-100: as with every layer (top) and as (bottom).

Figure A.23: Null controls for HCL-FF on CIFAR-100: as with every layer (top) and as (bottom).

Figure A.24: Null controls for VGG16 on Tiny ImageNet: as with every layer (top) and as (bottom).

Figure A.25: Null controls for SCFF on Tiny ImageNet: as with every layer (top) and as (bottom).

Figure A.26: Null controls for Trifecta on Tiny ImageNet: as with every layer (top) and as (bottom).

Figure A.27: Null controls for ASGE on Tiny ImageNet: as with every layer (top) and as (bottom).

Figure A.28: Null controls for DeeperForward on Tiny ImageNet: as with every layer (top) and as (bottom).

Figure A.29: Null controls for Mono-Forward on Tiny ImageNet: as with every layer (top) and as (bottom).

Figure A.30: Null 1, agreement of the two probes for SCFF (BP on the SCFF architecture shares these untrained networks): BPI under the paper’s probe (A, seed 42) against BPI under a second probe (B, seed 43) fitted on the same untrained features, one panel per layer, test datapoints of the three untrained networks pooled. Log-count hexbins; white line: $y=x$ ; dotted lines: $\mathrm{BPI}=0$ . $r$ : Pearson correlation of the two BPIs; flips: share of test datapoints that the two probes classify differently.

Figure A.37: As , for VGG16 (the pretrained pipeline: CIFAR-10 upsampled to $224\times 224$ ).

Figure A.38: As , on CIFAR-10, with every layer read on its own (no concatenation), for one seed and the first six layers.

Figure A.39: As , on CIFAR-100 (without SCFF and DeeperForward, whose checkpoints are not available).

Figure A.40: As , on Tiny ImageNet (without SCFF and DeeperForward, whose checkpoints are not available).

Figure A.41: As , on ImageNet-1K: HFF-VGG16 (our training) and the pretrained VGG16, against their untrained networks (probes fitted on 100 training images per class).

Decision Boundary Analysis

See all 8 papers

Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant
May 7, 2026Amirhossein YousefiramandiNeural Network Training Dynamics Local Learning
Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses
May 20, 2025Nazmus Saadat As-Saquib, A N M Nafiz Abeer, Hung-Ta Chien +3Neural Network Optimization Local Learning
Texas A&M University, College Station, TX 77843 · Sandia National Laboratories, Livermore, CA 94550
The Price of Locality: Why Forward-Forward Underperforms Backpropagation?
Sep 27, 2026Zhaoxian Wu, Haichuan Liu, Tianyi ChenForward-Forward Learning Neural Network Optimization
Cornell Tech and Cornell University

May 7, 2026J/K move · Enter open · S save

Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant

Amirhossein Yousefiramandi

Neural Network Training Dynamics Local Learning Forward-Forward Learning

Open paper AskPDF arXiv

Forward-Forward (FF) training lets each layer learn from a local goodness criterion. In cumulative-goodness variants, later layers can inherit a task that earlier layers have partly separated. We formalize this as layer free-riding: under the softplus FF criterion, the class-discrimination gradient reaching block

d

decays exponentially with the positive margin accumulated by earlier blocks (Theorem 3.1); a squared-hinge barrier reproduces the pathology. History-free and hardness-gated repairs raise deeper-layer separation by up to

5\times

on CIFAR-100 and

45\times

in an 8-block CIFAR-10 model, yet these and a depth-scaled auxiliary term move accuracy by under one percentage point among non-degenerate variants; on Tiny ImageNet, a harder cross-dataset check of the selected configuration, layer health likewise does not track accuracy. A theory-derived attenuation-compensated objective raises deepest-block separation

5.9\times

at a consistent cost in the FF classifier's own accuracy that a probe trained on the frozen features recovers; in a smaller CIFAR-10 control, most of its separation gain comes from dropping the depth-scaled auxiliary term it replaces. Accuracy invariance is a property of the deployed sum-based scoring rule under redistribution (Proposition 3.2) and, empirically, of the structured trained readouts we test, not of what the network learns: retrained variants disagree on 13-26% of test predictions. Architecture and augmentation matter far more than the non-degenerate training rules studied. Cumulative free-riding is therefore a real and repairable pathology, but for the softplus and squared-hinge objectives, architectures, and datasets we study, it is not the dominant factor limiting accuracy.

Abstract

Decision Boundary Analysis

Explore similar work

Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant

Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses

The Price of Locality: Why Forward-Forward Underperforms Backpropagation?