Trust Your Targets: Is Target Propagation Block-Diagonal Gauss-Newton?

Giuseppe Castiglione; Christopher Buckley; Ivor Simpson

cs.LGOct 19, 2026

Trust Your Targets: Is Target Propagation Block-Diagonal Gauss-Newton?

Authors: Giuseppe Castiglione, Christopher Buckley, Ivor Simpson

Organizations: Department of Informatics University of Sussex

Abstract

Target propagation (TP) is a backpropagation alternative in which local targets carry the global learning signal. Prior works claim that TP performs Gauss-Newton (GN) optimisation on these targets, but the correspondence has only been established under unrealistic assumptions where hitting them exactly is trivial. Practical variants instead descend on a least-squares problem via a single gradient step. In this paper, we rederive TP from a trust region under an implicit, previously unexamined norm, and introduce CGTP (Conjugate Gradient Target Propagation), which uses conjugate gradients to solve the layer-local subproblems in parallel. A single adaptive parameter jointly controls this trust region and a matching bound on the weight update; KFAC emerges exactly as the special case where the per-sample Jacobians are replaced by their batch mean, giving its factored damping a variational grounding. On MNIST and Fashion-MNIST MLPs, CGTP matches the accuracy of ADAM and KFAC while needing orders of magnitude fewer outer iterations than ADAM (about 10x fewer backward-pass-equivalents overall), while remaining stable across depth.

Figures & tables

Assumption	Relaxed in
$\mathfrak{A}$ 1 : Infinitesimal step size $\beta$	Section 3 : Trust-region radius
$\mathfrak{A}$ 2 : $\ell_{2}$ regression loss	Section 3.1 : Generalized surrogate loss
$\mathfrak{A}$ 3 : Single-layer updates	Section 3.2 : Norm-penalties
$\mathfrak{A}$ 4 : Strictly invertible layers	Section 3.2 : Norm-penalties, Section 4.2 : Simplicial decoupling
$\mathfrak{A}$ 5 : Mini-batch of size 1	Section 4.1 : Constrained activation space

Table 1: Assumptions invoked by prior proofs of the TP-GN equivalence

	F-MNIST				MNIST
alg	final err	outer @ 0.13	/it	total	final err	outer @ 0.05	/it	total
ADAM	$.111\pm.000$	$25{,}667\pm 4509$	1	$25.7$ k	$.019\pm.000$	$7333\pm 577$	1	$7.3$ k
KFAC	$.108\pm.002$	$2700\pm 500$	1	$2.7$ k	$.018\pm.001$	$467\pm 58$	1	$467$
DRL-lin.	$.111\pm.004$	N/A	N/A	N/A	$.020\pm.008$	N/A	N/A	N/A
TJML	$.114\pm.001$	$1468\pm 724$	1	$1.5$ k	$.018\pm.001$	$1377\pm 756$	1	$1.4$ k
TPRI	$.152\pm.004$	$\times$	$\times$	$\times$	$.061\pm.003$	$\times$	$\times$	$\times$

Table 2: Error and convergence cost, in backward-pass equivalents. Outer iterations are counted to the error threshold noted in each column header.

\times

denotes failure to reach threshold. N/A indicates result is not reported by source.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Batch size	Peak VRAM (mb)	FLOPs / step	Total Wall Clock (10 epochs)
BP+ADAM	0.01	103.49	1.08 GFLOP	6m 11s
KFAC	0.1	660.40	93.8 GFLOP	6m 24s
CGTP	0.1	3210.10	1.16 TFLOP	6m 47s

Appendix

Table 3: Resource comparison between BP+ADAM and CGTP

Preconditionr	Simplex Scheme	Test Score	[email protected]
False	Large $\lambda$	0.1085 ± 0.0016	6.6667 ± 0.5774
False	Uniform	0.1126 ± 0.0019	9.6667 ± 2.0817
False	Unit $\|\|\Delta z\|\|$	0.1107 ± 0.0008	12.6667 ± 2.0817
True	Large $\lambda$	0.1108 ± 0.0014	7.6667 ± 0.5774
True	Uniform	0.1240 ± 0.0027	9.0000 ± 1.7321
True	Unit $\|\|\Delta z\|\|$	0.1188 ± 0.0032	7.3333 ± 1.5275

Appendix

Table 4: Numerical comparison of the test error and number of iterations to reach a given test error for the different simplex schemes. Hyperparameters for each algorithm were optimised for validation error.

Figure 1: Comparing convergence of different optimisers at batch size 0.1 (Fashion-MNIST). CGTP hits the 0.13 score milestone after 7 batches. BP hits it at 50 batches, and KFAC does not reach it.

Figure 2: Results of a grid search over max CG iterations and batch size on validation error (MNIST dataset). By far the batch size has the biggest impact on model performance, due to its importance in estimating curvature (training curves may be seen in Figure 4 ). However, even with a poor curvature estimate, increasing the max cg iter can improve performance - though its impact is diminished once sufficient batch size is reached.

Figure 3: A graph of the impact of network depth on test error (Fashion-MNIST). In comparison with methods such as TPRI, CGTP is stable across depth. This is likely due to the difference between the regularised composite Jacobian, and the product of regularised layer Jacobians.

Figure 4: Varying the batch size at fixed max cg iter (Fashion MNIST): We keep the max cg iter fixed at 100, and vary the batch size for CGTP. Larger batches result drastically accelerated learning, which is also reflected in the test error (with diminishing returns). This highlights a known phenomenon of second order optimizers, as they need larger batch sizes to get better second order estimates of the curvature.

Figure 5: Varying the max cg iter at fixed batch size (Fashion-MNIST): We keep the batch sized fixed at batch size 0.1 for CGTP. We see small improvements in terms of learning speed on the train set, but this does little for the test error.

Gauss-Newton Optimization

Gauss–Newton optimization uses residual Jacobians to build tractable curvature-aware updates for nonlinear least-squares problems and machine learning.

See all 14 papers

Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses
May 20, 2025Nazmus Saadat As-Saquib, A N M Nafiz Abeer, Hung-Ta Chien +3Neural Network Optimization Local Learning
Texas A&M University, College Station, TX 77843 · Sandia National Laboratories, Livermore, CA 94550
Error whitening: Why Gauss-Newton outperforms Newton
May 11, 2026Maricela Best McKay, Nathan P. Lawrence, Brian Wetton +1Gauss-Newton Optimization Deep Learning Optimization
University of British Columbia · University of California, Berkeley
Backpropagation-Free Trunk Training via the Split Forward Gradients
Jul 18, 2026Tian Qin, Wei-Min HuangBackpropagation Gradient Descent
Math department Lehigh University Bethlehem, PA 18015, USA

May 20, 2025J/K move · Enter open · S save

Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses

Nazmus Saadat As-Saquib, A N M Nafiz Abeer, Hung-Ta Chien, Byung-Jun Yoon+2

Texas A&M University, College Station, TX 77843 · Sandia National Laboratories, Livermore, CA 94550

Neural Network Optimization Local Learning

Open paper AskPDF arXiv

Training neural networks has traditionally relied on backpropagation (BP), a gradient-based algorithm that, despite its widespread success, suffers from key limitations in both biological and hardware perspectives. These include backward error propagation by symmetric weights, non-local credit assignment, and frozen activity during backward passes. We propose Forward Target Propagation (FTP), a biologically plausible and computationally efficient alternative that replaces the backward pass with a second forward pass. FTP estimates layerwise targets using only feedforward computations, eliminating the need for symmetric feedback weights or learnable inverse functions, hence enabling modular and local learning. We evaluate FTP on fully connected networks, CNNs, and RNNs, demonstrating accuracies competitive with BP on MNIST, CIFAR10, and CIFAR100, as well as effective modeling of long-term dependencies in sequential tasks. Moreover, FTP outperforms BP under quantized low-precision and emerging hardware constraints while also demonstrating substantial efficiency gains over other biologically inspired methods such as target propagation variants and forward-only learning algorithms. With its minimal computational overhead, forward-only nature, and hardware compatibility, FTP provides a promising direction for energy-efficient on-device learning and neuromorphic computing.

Abstract

Gauss-Newton Optimization

Explore similar work

Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses

Error whitening: Why Gauss-Newton outperforms Newton

Backpropagation-Free Trunk Training via the Split Forward Gradients