LocalProp: Neuro-Localized Memory-Efficient Backpropagation

Diana-Nicoleta Grigore; Iuliana Georgescu; Radu Tudor Ionescu

cs.AISep 26, 2026

LocalProp: Neuro-Localized Memory-Efficient Backpropagation

Authors: Diana-Nicoleta Grigore, Iuliana Georgescu, Radu Tudor Ionescu

Organizations: Department of Computer Science University of Bucharest, Romania

Abstract

The current deep learning training paradigm employs end-to-end backpropagation, regardless of the training stage, i.e. pre-training or fine-tuning. However, backpropagating through the entire model is neither biologically plausible nor memory efficient, since learning inside the brain is highly localized. Therefore, we propose LocalProp, a training procedure that locally updates the weights of a model. Our neuro-localized weight updates follow the "pre-training then fine-tuning" paradigm, where the pre-training is based on I-JEPA. After locally updating the weights, a pruning operation is performed, followed by a short final fine-tuning phase. Pruning helps by sending the learning signal from higher blocks to lower blocks. We perform experiments on several datasets, including large-scale benchmarks such as ImageNet, and empirically show that LocalProp reaches good performance at a fraction of GPU peak memory. By varying the number of jointly optimized blocks, we identify gradient-propagation span as a practical control over the accuracy-memory trade-off.

Figures & tables

Figure 3: Overview of the LocalProp training pipeline. Our method applies local optimization throughout I-JEPA pre-training, supervised fine-tuning, structured pruning, and local recovery. Only the selected block is optimized and scored at each step, avoiding end-to-end backpropagation. During pruning, downstream task performance is used to score lower-level components, allowing information from higher blocks to guide structural decisions without backpropagation. Best viewed in color.

Setting	Local I-JEPA	Pruning	Recovery	Param. factor	ViT-S			ResNet-50
Setting	Local I-JEPA	Pruning	Recovery	Param. factor	CIFAR-10	CIFAR-100	ImageNet-1K	CIFAR-10	CIFAR-100	ImageNet-1K
Local FT (scratch)	✗	✗	✗	$1.2\times$	68.63	42.53	33.29	87.75	65.31	47.36
LocalProp (no pruning)	✓	✗	✗	$1.2\times$	76.69	50.53	37.12	89.79	67.07	55.75
LocalProp (no recovery)	✓	✓	✗	$1.0\times$	52.21	27.62	24.96	88.23	52.94	37.83
LocalProp	✓	✓	✓	$1.0\times$	75.40	47.48	34.81	88.84	67.26	52.19
Local I-JEPA probe	✓	✗	✗	$1.2\times$	49.46	19.16	8.27	32.80	11.09	11.89

Table 1: Effect of LocalProp training phases across architectures and datasets. Top-1 accuracy on CIFAR-10, CIFAR-100, and ImageNet-1K for ViT-S and ResNet-50 under multiple configurations of local pre-training, fine-tuning, pruning, and recovery. Reduced-size variants use the same parameter budget as the pruned models. Local pre-training improves subsequent local fine-tuning, while recovery restores much of the performance lost during pruning.

Input	Setting	Peak memory	Memory reduction	Top-1 accuracy
CIFAR-10	Full	3.48 GiB	0.0%	87.29%
CIFAR-10	Local (1 block)	0.69 GiB	80.2%	76.69%
ImageNet-1K	Full	3.40 GiB	0.0%	76.87%
ImageNet-1K	Local (1 block)	0.74 GiB	78.2%	37.12%

Table 2: Peak allocated memory for ViT-S is measured on an RTX 3090. Full updates backpropagate through the complete encoder, whereas local updates optimize one transformer block per step.

Method	Parameter factor	ResNet-50			ViT-S
Method	Parameter factor	CIFAR-10	CIFAR-100	ImageNet	CIFAR-10	CIFAR-100	ImageNet
LocalProp (no pruning)	$1.2\times$	89.79	67.07	55.75	76.69	50.53	37.12
LocalProp	$1.0\times$	88.84	67.26	52.19	75.40	47.48	34.81
NoProp ( Li et al., 2026 )	$1.2\times$	83.91	52.02	EB	NV	NV	NV
Greedy InfoMax ( Löwe et al., 2019 )	$1.2\times$	72.24	41.02	3.11	42.58	15.55	18.81
FF ( Papachristodoulou et al., 2024 ; Sarode et al., 2026 )	$1.2\times$	78.11	51.23	25.00	NV	NV	NV

Table 3: Comparison with alternatives to full backpropagation. Top-1 accuracy of LocalProp, NoProp, Greedy InfoMax, Forward-Forward, and DGL on CIFAR-10, CIFAR-100, and ImageNet-1K using ResNet-50 and ViT-S backbones. Results unavailable in the cited works are obtained from our reproductions. NV (no variant) denotes a backbone variant unavailable in the evaluated implementation. EB (exceeding budget) denotes a configuration whose full-run training time exceeded our per-run budget of 72 GPU-hours (3 days) on an NVIDIA RTX 3090.

Table 6

Figure 4: Feature visualizations on an ImageNet validation image. Feature magnitude is shown at transformer blocks $0$ , $5$ , and $11$ for local I-JEPA, LocalProp, pruned LocalProp, and the same-mask architecture trained from scratch. Colors are jointly normalized across methods within each block row, and the dashed contour marks the same object region in all final maps. LocalProp develops a clearer object-aligned late-layer response than the scratch model, which is largely preserved after pruning.

Method	#blocks	TFLOPs	GiB	GPU-h
Local FT	$1/12$	$0.195$	$0.74$	$27.8$
Full FT	$12/12$	$2.340$	$3.40$	$75.1$
Reduction	$12.0{\times}$	$12.0{\times}$	$4.6{\times}$	$47.3$ saved

Table 6: Training cost of block-wise local fine-tuning vs. end-to-end fine-tuning for ViT-S/16. Backward operations are reported per optimization step, and VRAM is the peak allocated memory on an RTX 3090. GPU-hours are measured for block-wise ImageNet fine-tuning and full fine-tuning over 120 epochs.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	ViT-S/16	ResNet-50
Pre-training epochs	600	200
Optimizer	AdamW	AdamW
Learning rate	$10^{-4}$	$[1,3,.75]\!\times\!10^{-4}$
Warmup	40 ep.	5 ep.
Weight decay	$0.04$	$10^{-4}$
Batch size	4096 (effective)	128

Appendix

Table 7: Key hyperparameters for Local I-JEPA pre-training. The active transformer block or ResNet group is selected in round-robin order. For ViT-S, the learning-rate entry denotes the initial learning rate before warmup and cosine decay. The three ResNet-50 rates correspond to group

0

, groups

1

–

12

, and group

13

, respectively.

Pruning order	Post-pruning	After recovery
Deepest to shallowest (main)	88.21%	88.84%
Shallowest to deepest	88.04%	89.13%

Appendix

Table 8: Effect of pruning order on ResNet-50. CIFAR-10 top-1 accuracy after channel pruning and recovery. Both traversal orders use the same starting checkpoint, pruning budget, scoring procedure, and structured masked constraints, removing

24.29\%

of the parameters. The main configuration processes layers from deepest to shallowest.

Panel of Figure 5 — Figure 5: (a) Cosine similarity between gradients induced by the block-local objective and those received by the same block under full backpropagation. Local gradients across depth have a mean norm ratio of $1.04\pm 0.02$ . (b) Each cell shows the increase in final classification loss after temporarily ablating one attention head. White markers indicate heads retained under an example partial-retention budget. Best viewed in color.

Model	Dataset(s)	Epochs	Batch size	Learning rate	Weight decay	LR schedule	Local Updates
ViT-S/16	CIFAR-10, CIFAR-100, ImageNet-1K	120	64	$5\times 10^{-5}$	$0.05$	Cosine	8 updates / block
ResNet-50	CIFAR-10, CIFAR-100	500	64	$3\times 10^{-4}$	$0.01$	Constant	all 14 groups
ResNet-50	ImageNet-1K	500	224	$3\times 10^{-4}$	$0.01$	Constant	all 14 groups

Appendix

Table 9: Key hyperparameters for local supervised fine-tuning. All runs use AdamW and cross-entropy with label smoothing

0.1

Hyperparameter	ViT-S/16	ResNet-50
Hyperparameter	ViT-S/16	CIFAR	ImageNet
Pruning unit	Attention heads	Output channels	Output channels
Pruning target	48/72 heads removed	$1/3$ of each width removed
Scoring budget (batches)	8	8	8
Recovery budget (epochs)	10	10	10
Recovery batch size	64	64	224

Appendix

Table 10: Key pruning and recovery hyperparameters. Structured masks are enforced throughout local recovery by zeroing the corresponding attention projections for ViT-S and convolutional filters and normalization parameters for ResNet-50.

Setting	Seed 0	Seed 1	Seed 42	$\mu\pm\sigma$
LocalProp	$76.69$	$77.17$	$76.76$	$76.87\pm 0.26$
Pruned	$52.21$	$53.86$	$52.36$	$52.81\pm 0.91$
Recovered	$75.40$	$75.93$	$75.40$	$75.58\pm 0.31$
Same-mask Scratch	$66.85$	$66.46$	$66.68$	$66.66\pm 0.20$

Appendix

Table 11: Top-1 accuracy across three training seeds. Variability is reported using the sample standard deviation.

Panel of Figure 6 — Figure 6: Additional layerwise feature-magnitude visualizations. Columns show the input, local I-JEPA, LocalProp, pruned LocalProp and the same-mask architecture trained from scratch. Rows show transformer blocks $0$ , $5$ , and $11$ for three ImageNet-1K validation examples. Feature magnitudes are jointly normalized across methods, and dashed contours indicate the same object region. Best viewed in color.

Local Learning

Local learning develops neural training rules that use nearby activity or layerwise objectives instead of global backpropagated errors.

See all 50 papers

Local proxies for memory-efficient training
Oct 16, 2026Wu S. Zihan, Brian A. Yuan, Guillaume BellecLocal Learning Memory-Efficient Training
Machine Learning Research TU Wien Vienna, Austria
Local Pheromone Network: Sparse Local Learning with Multi-Scale Synaptic Trails, Consolidation, and Replay
Jun 22, 2026Xingcheng Fu, Xianjun Chen, Zhihao LiHebbian Learning Continual Learning
Breaking chains with trees: Deep learning with $\mathcal{O}(\log N)$ parallel time complexity
Jun 19, 2026Neeraj Mohan Sushma, Aditya Nagarsekar, Cabrel Teguemne Fokam +4Deep Learning Optimization Neural Network Optimization
Center for Cognitive Interaction Technology, Bielefeld University, Germany · Ruhr Universität Bochum, Germany · Birla Institute of Technology and Science, Pilani, Goa Campus +1

Oct 16, 2026J/K move · Enter open · S save

Local proxies for memory-efficient training

Wu S. Zihan, Brian A. Yuan, Guillaume Bellec

Machine Learning Research TU Wien Vienna, Austria

Local Learning Memory-Efficient Training Memory-Efficient Fine-Tuning

Open paper AskPDF arXiv

Local learning algorithms can reshape resource requirements of artificial intelligence. The memory bottleneck of standard end-to-end backpropagation lies in the need to store all network activations until the loss gradient has been propagated back through. Local learning avoids this, but typically at a cost in accuracy. We study how to achieve accurate and memory efficient local learning in transformers. For each transformer block, we train a local proxy network to predict the global network output via distillation. Gradients can be independently propagated through this local proxy while estimating well the backpropagation gradients. In theory, this approach is exact for deep linear networks; in non-linear residual models, the gradient error can be bounded when using the appropriate residual proxy. We verify this numerically and demonstrate that optimizing the distillation loss improves gradient estimation. Evaluating the performance of proxy propagation on CIFAR-100 in vision transformers, we find that it achieves a better accuracy/memory trade-off compared to other local algorithms. Finally, we show that the method is further relevant for fine-tuning large language models on a small memory budget. Our method trains faster and achieves higher performance than memory-efficient zeroth-order learning algorithms with comparable memory cost.

Abstract

Local Learning

Explore similar work

Local proxies for memory-efficient training

Local Pheromone Network: Sparse Local Learning with Multi-Scale Synaptic Trails, Consolidation, and Replay

Breaking chains with trees: Deep learning with O(log⁡N)\mathcal{O}(\log N)O(logN) parallel time complexity

Breaking chains with trees: Deep learning with $\mathcal{O}(\log N)$ parallel time complexity