Local Learning

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 34

Sep 28, 2026cs.LG

SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learning

Large language models are trained with backpropagation, whose global gradient coordinates all layers but forces each to hold its activations and wait for the gradient to pass back through every deeper layer. Conventional local learning removes this update locking by training each module to predict the target through its own readout, but has not scaled to billion-parameter pretraining. We identify these private readouts as a key weakness, since they leave each module without information from deeper modules. We propose Shared-Output LOcal learning (SOLO), which replaces them with a shared, read-only copy of the final module's readout, the only one trained on the output of the whole network. Taken from the previous step, the copy transmits information from the final module without passing gradients between modules or reintroducing update locking. SOLO approaches backpropagation on Transformers of 340M to 2B parameters pretrained on 15B tokens, staying within one point in average zero-shot accuracy with a perplexity gap that narrows with scale. Readout ablations attribute SOLO's improvement over private readouts to sharing. Without update locking, each of p pipeline stages holds activations for O(1) micro-batches instead of O(p). The freed memory permits larger micro-batches, which reach up to 1.44x the best measured throughput of pipeline backpropagation on the same partition. To our knowledge, SOLO is the first local learning method to show such memory and throughput gains in billion-parameter language-model pretraining. Local learning thus becomes a practical alternative to backpropagation for large-scale pretraining.

Bojian Yin, Shurong Wang, Yuqi Pan +1

Institute of Automation, Chinese Academy of Sciences

Language Model Pretraining Efficient Language Model Training Local Learning

Sep 14, 2026physics.data-an

Online local learning for generative thermodynamic computing

Generative thermodynamic computers turn thermal noise into structured data through Langevin dynamics. We train these systems with a local update at each integration step. The reverse-path Onsager-Machlup objective yields a coupling gradient that is a symmetric sum of local residual-state correlations. We apply this gradient immediately rather than accumulating it over a full trajectory. In digital simulations using MNIST prototypes, online and trajectory-batch training reach similar validation losses on fixed noising paths. Models trained online release less heat on average in all five independently seeded pairs, with both models' parameters held fixed during sampling. Auxiliary classifier and nearest-prototype measures change modestly, while pairwise diversity decreases. The response to noise depends strongly on where the errors enter: independent zero-mean errors in the formed updates produce little heat change over a finite range of noise amplitudes, whereas residual offset and temporal correlation have much larger effects. Storing trained couplings requires substantially less precision than resolving deterministic updates during training. Together, these results establish a local online training method and show how update timing, noise structure, and precision affect generative thermodynamic computing.

Huilin Wang, Weibing Deng

Key Laboratory of Quark and Lepton Physics (MOE) and Institute of Particle Physics, Central China Normal University, Wuhan 430079, China

Thermodynamic Computing Local Learning Online Learning

Sep 1, 2026cs.LG

A Study of Hidden-State Optimization Order in Predictive Coding Networks

Local learning methods offer an alternative to end-to-end backpropagation, but their unstructured local objectives can produce weak feature learning in deep networks. We study whether the order of hidden-state optimization can address this limitation. We propose a boundary-first inference schedule that partitions a model into chunks, first coordinates hidden states at chunk boundaries, and then refines representations within each chunk. We instantiate this schedule in predictive coding networks (PCNs), a local-learning framework in which hidden activities and prediction errors are explicitly exposed during inference. On CIFAR-10, the resulting boundary-first predictive-coding instantiation improves accuracy over standard predictive coding by

9.77\%

under a standard parametrization and by

5.51\%

under a

μ

-parametrization. Diagnostic analyses further show more non-trivial early-layer updates, lower initial-to-final CKA, and more diverse layerwise gradients, consistent with stronger feature learning. These results support boundary-first, chunk-based inference as a practical design principle for predictive-coding training and motivate its study in broader local-learning systems.

Xueyuan Li, Danilo Vasconcellos Vargas

Predictive Coding Neural Network Optimization Local Learning

Jul 30, 2026cond-mat.soft

Conservation laws determine what physical learning remembers

Physical learning rules such as equilibrium propagation (EP), coupled learning (CL), and adjoint coupled learning (AL) train resistive networks through local measurements. In the small-nudge limit EP and CL exactly conserve the conductance mass K = (1/2) sum_e kappa_e^2, a property that stabilizes training. We show that conservation also governs the inductive bias of these rules. For a single output we prove that EP and CL are trajectory equivalent, so single-output experiments cannot distinguish what the two rules learn. We prove that AL does not conserve the mass but dissipates it at exactly twice its own loss. In linear circuits we prove that the conserved mass has no functional consequence: all three vector fields are homogeneous in the conductances, so the selected solution is independent of the initialization scale. Fixed nonlinear elements break this protection. In diode circuits the learned input-output function depends on the initialization scale by up to about forty percent, an effect absent in linear controls, and the conservative rules retain this memory permanently while the dissipative rule partially erases it. At matched training loss the dissipative rule typically generalizes worse than the conservative rules, although it reaches low training loss faster; the penalty correlates with the mass dissipated en route and fades in larger circuits, where little mass is lost. The conservation structure of a local learning rule thus sets its initialization memory, its training speed, and, where dissipation is appreciable, its generalization; it should be treated as a design parameter of physical learning machines.

Bijaya Dangol

1Independent researcher

Equilibrium Propagation Neural Network Training Dynamics Local Learning

Jul 29, 2026cs.LG

The Art of Not Forgetting A Local Learning Architecture for Continual Learning

We introduce CMP (Cognitive Memory Primitive), a continual-learning architecture that repre?sents inputs as sparse relational codes, stores them in a two-tier competitive memory, and learns through local updates without end-to-end backpropagation through its feature-generating system. We investigate whether combining sparse representations, local learning, and persistent memory can reduce catastrophic forgetting relative to conventional backpropagation-based continual?learning approaches. On a controlled domain-incremental byte-level language modeling protocol, CMP demonstrates substantially lower backward transfer than a parameter-matched Trans?former trained with online Elastic Weight Consolidation (EWC). Across a three-seed replicated 15-domain experiment, CMP exhibits stable forgetting behavior, while separate head-to-head comparisons and domain-order analyses show consistently lower forgetting than the evaluated Transformer baseline under the reported experimental settings. We report these findings alongside a substantial single-domain accuracy gap relative to the Transformer, a null result on a vision benchmark, and a documented failure to combine CMP with an independent accuracy-improving mechanism, reflecting our commitment to reporting both positive and negative outcomes. These results suggest that the combination of sparse representations, local learning, and persistent memory is a promising direction for continual learning, while motivating further investigation into the respective roles of learning rules, representations, and architectural design in mitigating catastrophic forgetting.

Ashmith Atmuri, Yashaswini Rao Bhogarajula

Arkadhi Research

Memory-Augmented Neural Networks Continual Learning Domain-Incremental Learning Incremental Learning Local Learning Catastrophic Forgetting

Jul 29, 2026cs.NE

Reconstructing Backpropagation from Forward Fluctuations in Noise-modulated Neural Networks

A Noise-modulated Neural Network (NNN) learns and infers only in the presence of noise, treating noise as a computational resource rather than a disturbance. The noise lets it learn efficiently by backpropagation while transmitting spike-like signals, but backpropagation needs a reverse path through transposed weights, the weight transport problem, which undermines biological and neuromorphic plausibility. Forward-only alternatives typically substitute a different objective or fixed random feedback, sacrificing stability and accuracy. We show that backpropagation itself can be reconstructed in the NNN from forward-pass statistics alone: a weight mirror estimates each weight matrix from the covariance between a previous-layer unit's output and the next-layer unit's input, and combining it with local differential estimation inside the units propagates the output error recursively along the computational graph, with no transposed-weight readout and no backward data path. The resulting gradient is empirically near-unbiased, and with local per-weight Adam updates it matches the final accuracy of backpropagation on simple regression tasks. With uniformly distributed noise, the local operations reduce to polynomials and comparators, making the whole system, learning rule included, well suited to digital circuits. Thus, in the NNN, noise is a resource not only for inference but also for reconstructing backpropagation.

Shuhei Ikemoto

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, Kitakyushu, Japan

Backpropagation Neuromorphic Computing Local Learning

Jul 20, 2026cs.LG

The Art of Not Forgetting

We introduce CMP (Cognitive Memory Primitive), an architecture that represents inputs as sparse relational codes, stores them in a two-tier competitive memory, and learns entirely through local, gradient-free updates, with no backpropagation anywhere in the network. We use this architecture to test a specific hypothesis: that catastrophic forgetting, usually treated as a training-time defect to be patched with replay or regularization, is instead a structural consequence of how backpropagation assigns credit and that a learning rule that is local and sparse by construction should resist it without a patch. On a controlled domain-incremental protocol across 15 text domains, three-seed replicated, CMP's backward transfer is 15-19x better than a matched-size Transformer trained with online EWC, and the result survives a domain-order control (reported as a range, +0.24 to +0.44, rather than a single figure). We report this alongside a real, substantial accuracy gap versus the Transformer baseline, a null result on a recognized vision benchmark, and a diagnosed, unresolved failure attempting to combine this architecture with a separate mechanism that improves raw accuracy, disclosed because an honest negative result is more useful than an omitted one. The central claim is narrow and falsifiable: local, sparse, non-backpropagation learning measurably resists catastrophic forgetting better than backpropagation with its standard fix, under conditions we state precisely.

Ashmith Atmuri, Akshay Kumar, Yashaswini Rao Bhogarajula

Arkadhi Labs

Memory-Augmented Neural Networks Domain-Incremental Learning Incremental Learning Local Learning Catastrophic Forgetting

Jul 15, 2026cs.LG

Weight Feedback Computes the Jacobian Transpose Locally in Modern Deep Networks

Predictive Coding (PC) offers a biologically motivated alternative to backpropagation via local weight updates, yet routing error between layers still relies on an autograd Jacobian-transpose (

J^\top

) product - the last non-local operation in PC. We show that this dependency is largely avoidable. For any layer

f(x)=\mathrm{Act}(\mathrm{Norm}(L(x)))

with frozen normalization statistics, the exact

J^\top

factors into three locally available terms,

J^\top v = L^\top(s \odot σ'(z) \odot v)

, where

σ'

is the activation derivative,

z

is the pre-activation, and

s=γ/σ_{\mathrm{run}}

is the normalization gain. Prior weight-feedback methods omitted both corrections; restoring them closes the transport gap for this layer class. Locality here holds up to three assumptions, which we state upfront: weight symmetry (

L^\top

mirrors the forward operator, as assumed by all PC), a soft spectral-norm control that is not synapse-local, and a nearest-neighbour approximation for MaxPool. Substituting the identity into PC yields WF-Act-PC, which removes the autograd backward pass from error transport. On CIFAR-10/100 (50 epochs, 5 seeds), WF-Act-PC is the only PC method whose accuracy improves with depth, surpassing iPC - the strongest classical PC baseline - by 2.7-22.3 pp on CIFAR-10. With both methods tuned per architecture, it matches or exceeds a comparably-tuned backpropagation baseline on the deeper CIFAR-10 architectures (VGG-9: 93.57% vs. 92.43%; ResNet-18: 92.76% vs. 91.54%) and on the harder Tiny-ImageNet benchmark, while trailing tuned BP on the deeper CIFAR-100 VGG cells. Our WF-Act-PC implementation is publicly available at https://github.com/jlshen025/pcax

Junlong Shen, Xingyu Li

Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada · Alberta Machine Intelligence Institute (Amii), Edmonton, Canada

Predictive Coding Local Learning

Jul 8, 2026cs.LG

Architecture Generalization with MetaNCA

Self-organization is an emergent property of life, driven by the collective behavior of individual components acting on local information. Biological neurons, through local interactions transmitted through synapses, are able to learn efficiently and can adapt their connections over an organism's lifespan. Motivated by these desirable properties of adaptability and local interaction, neural cellular automata (NCA) models have been successful at learning morphogenesis solely through local update rules, demonstrating stability over many updates and robustness to perturbations. In this work, we introduce Meta Neural Cellular Automata (MetaNCA), a framework that learns local rules which self-organize the weights of artificial neural networks. A learned rule network iteratively updates the weights of a task network using only local interactions on the computation graph. We propose a novel Weight Transformer architecture for the local rule network, which uses linear attention to aggregate signals from neighboring weights and hidden states. Once trained, the rule network generates task networks of diverse architectures without backpropagation. We show that MetaNCA generates weights for feedforward MLPs, CNNs, and ResNets on MNIST and CIFAR-100, scaling to networks of 2 million parameters. We further show that MetaNCA generalizes to architectures not seen during meta-training, and that architectural diversity in the training phase strengthens this generalization.

Meet Barot, Daniel Berenberg, Sina Khajehabdollahi

Mythos Scientific, New York, USA · Independent Scholar

Neural Cellular Automata Neural Network Generalization Meta-Learning Local Learning

Jun 22, 2026cs.NE

Local Pheromone Network: Sparse Local Learning with Multi-Scale Synaptic Trails, Consolidation, and Replay

Backpropagation-trained dense neural networks are powerful function approximators, but they couple learning across many parameters and can overwrite previous associations when tasks conflict. This paper describes Local Pheromone Network, a small research prototype for sparse, local, manually updated neural networks. In Local Pheromone Network, each output unit reads only a fixed local neighborhood of input units subject to geometric distance and molecular-tag compatibility. Each synapse stores a weight, a short-term pheromone trace, a long-term pheromone trace, and an optional consolidation state. Training does not call automatic differentiation. Instead, every layer performs a pheromone-weighted Hebbian-style update on a budgeted subset of local synapses selected from local error and co-activity. The update budget adapts online: it shrinks when loss improves and expands toward recently active neighborhoods when loss worsens. Optional mechanisms add structural plasticity, local replay, output masks for partitioned learning, and a target-free local contrastive step. We present the implementation, learning rule, and preliminary experiments on synthetic regression, partitioned memory, conflicting memory, consolidated conflict, structural plasticity, replay, and a synthetic long-context hybrid memory task. The prototype learns local linear rules, preserves partitioned memories through tags and masks, reduces forgetting under consolidation, and uses replay under conflict.

Xingcheng Fu, Xianjun Chen, Zhihao Li

Hebbian Learning Continual Learning Sparse Neural Networks Local Learning

Jun 22, 2026cs.LG

Error Highways: Scaling Predictive Coding to Very Deep Networks

Predictive coding networks (PCNs) offer a biologically-plausible, local-learning alternative to back-propagation of errors (backprop). Nevertheless, they have remained largely confined to shallow architectures and evaluated on simple machine intelligence benchmarks. A central obstacle to scaling PCNs is that the learning signal decays rapidly as it propagates away from the clamped boundaries, leaving interior layers effectively unchanged. To directly counter this problem, we propose highway error propagation (HEP), a scheme that augments the free energy function underlying predictive coding (PC) by altering its neural structure with feedback matrices

V_{L\to i}

that couple selected hidden states directly to the clamped output error. Since this coupling is linear in the hidden state, the highway pathway delivers a correction at every inference step whose magnitude is independent of depth, in contrast to vanilla PC where the output error reaches the

i

-th hidden layer with attenuation that decays exponentially in depth. This bypasses the Jacobian chain while preserving the local PC synaptic update rule. On MNIST and Fashion-MNIST, we show that HEP effectively trains MLPs of up to 128 layers with accuracy that is robust with respect to depth.

Amirhossein Mohammadi, Alexander G. Ororbia

SingularityNET Zug, Switzerland · Rochester Institute of Technology Rochester, NY, USA

Predictive Coding Local Learning

Jun 21, 2026cs.LG

Noise-Debiased Thermodynamic Variance for Local Learning Coefficient Probes

Local learning coefficient (LLC) probes offer a singularity-aware view of neural-network training, but mean-energy methods require a local loss baseline that is ambiguous at transient checkpoints. Thermodynamic variance avoids this input; under mini-batch evaluation, however, direct variance mixes cross-state loss fluctuations with same-state noise. We operationalize this route with the \emph{Shift-Invariant Variance Estimator} (SIVE), which estimates and subtracts the latter component using repeated evaluations. Conditional on any fixed retained path, unclipped SIVE is unbiased for noiseless path variance without requiring MCMC stationarity. The finite-scale diagnostic remains indexed by localization scale

h

---even a locally linear loss has tether-dependent variance---while interpretation as a Real Log Canonical Threshold (RLCT) requires additional stationary low-temperature conditions. Toy experiments recover calibrated finite-scale targets. At the primary localization scale, all five MNIST MLP trajectories exhibit a mid-training trough followed by a rebound in SIVE, while Raw Variance decreases from Epoch 40 to 100 in every trajectory. Across four localization scales, the joint early-drop/late-rise criterion is met in 19 of 20 trajectory--scale pairs. At Epoch 40, the estimated observation-noise correction accounts for

77.5\%

of Raw Variance. Same-state debiasing thus reveals a reproducible turning structure masked by time-varying observation noise.

Yingjia Cai

Singular Learning Theory Neural Network Training Dynamics Statistical Learning Theory Local Learning

Jun 19, 2026cs.LG

Breaking chains with trees: Deep learning with $\mathcal{O}(\log N)$ parallel time complexity

Modern deep neural network architectures are trained via backpropagation, which requires errors to be sequentially propagated through all layers before parameters can be updated. This introduces two limitations: locking, where layer-wise updates are strictly interdependent and cannot proceed in parallel, and the weight transport problem, which requires symmetric forward and backward pathways for exact gradient computation. These constraints restrict parallelism, increase memory and communication overhead, and pose challenges for scalable learning. In this work, we propose Hierarchical Block-Local Learning (HBLL), a framework that decomposes deep neural networks into hierarchically linked blocks trained using local learning objectives derived from variational principles, eliminating the need for full end-to-end backpropagation while maintaining effective information propagation across the network. HBLL is the first algorithm that is able to train deep neural networks in

\mathcal{O}(\log N)

parallel time complexity, where

N

is the number of network layers. We show that HBLL implicitly defines a family of subnetworks corresponding to different hierarchical paths, enabling flexible inference with different effective numbers of layers. We evaluate HBLL on a set of challenging vision and language modeling tasks, achieving competitive performance. We also extend HBLL to recurrent sequence architectures, applying to settings that otherwise rely on backpropagation through time.

Neeraj Mohan Sushma, Aditya Nagarsekar, Cabrel Teguemne Fokam +4

Center for Cognitive Interaction Technology, Bielefeld University, Germany · Ruhr Universität Bochum, Germany · Birla Institute of Technology and Science, Pilani, Goa Campus +1

Deep Learning Optimization Neural Network Optimization Local Learning

Jun 13, 2026math.OC

Coercivity and Local Convergence of Physical Learning in Linear Circuits

Physical learning methods train physical networks to perform computational tasks using only local update rules, exploiting the physics of the system to handle the global transfer of information. We provide the first local convergence analysis of three such methods -- Equilibrium Propagation (EP), Coupled Learning (CL), and a new method we call Adjoint Coupled Learning (AL) -- for linear circuits, in the limit of small-nudging for both discrete and continuous time. EP and AL perform gradient descent on a natural loss function, while CL follows modified dynamics with an additional cubic correction. Assuming the existence of a solution, we identify a coercivity condition, expressed as a rank condition on a matrix built from the network's incidence structure, under which the training loss decays exponentially and the parameters converge to the solution manifold. We show that coercivity can fail by exhibiting a kite circuit in which a symmetry causes the coercivity constant to degenerate on the solution manifold, but prove using Sard's theorem that such degeneracies are non-generic: coercivity holds at every point of the solution manifold for almost every choice of desired output.

Joshua A. McGinnis, Xinbo Li, Yoichiro Mori

Department of Mathematics, University of Pennsylvania, Philadelphia.

Equilibrium Propagation Optimization Convergence Analysis Local Learning

Jun 4, 2026cs.CV

Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training

Forward-Forward (FF) learning [Hinton, 2022] replaces backpropagation with strictly layer-local goodness updates. Recent FF-CNN work has narrowed the gap to BP on 32x32 benchmarks, raising the question of whether layer-local training is becoming a viable alternative at realistic scale. To probe this rigorously, we develop DTG-FF -- dynamic temperature goodness, decoupled normalization, and multi-layer fusion -- as an instrument that sets FF-family state of the art across nine real-data benchmarks at the time of our submission (91.8% CIFAR-10 and an FF baseline at ImageNet-100 224x224), and use it to audit how far layer-local training actually scales. (1) Real-data scaling. Under identical recipe and backbone, an architecture-matched BP-DeepSup baseline beats DTG-FF by 2.40/5.93 pp on CIFAR-10/CIFAR-100, and the gap widens with class count. At 224x224 the same instrument reaches only 49.4% on ImageNet-100, versus 75.6% even for a self-supervised BP-trained ResNet-50 with linear evaluation [Tian et al., 2020] -- exposing a real-data ceiling invisible at 32x32. (2) Synthetic vs. real K-conflict. DTG-FF increasingly outperforms BP as class count K grows on synthetic teacher-student tasks, yet on real images the FF-BP gap reverses sign and widens with K. A within-dataset CIFAR-100 coarse vs. fine probe isolates label-hierarchy from image distribution: synthetic K-sweeps confound output dimensionality with fine-grained discrimination difficulty and thereby overstate FF transferability. (3) Systems audit. FF can be implemented without storing depth-wide activations, but on commodity 8 GB hardware standard BP+gradient-accumulation reaches 4.18 GB / 157 imgs/s versus DTG-FF's 7.90 GB / 138 imgs/s, so a memory-based justification for FF at this scale is not supported under fair baselines.

Yucheng Chen

Amplimit Toronto, ON, Canada

Backpropagation Local Learning Forward-Forward Learning Image Classification

Jun 2, 2026cs.LG

FFR: Forward-Forward Learning for Regression

The Forward-Forward (FF) algorithm offers a computationally efficient and biologically plausible alternative to backpropagation (BP) by training neural networks through purely local, layer-wise optimization. However, FF is inherently designed for classification via contrastive positive-negative sample pairs, and extending it to regression poses fundamental challenges: continuous target space lacks natural "opposites" for contrastive learning, and the standard goodness function carries no information about target magnitude or ordering. We propose FFR (Forward-Forward for Regression), to our knowledge, the first framework to extend FF to real-world regression and demonstrate competitive performance across diverse realworld datasets. FFR introduces three key innovations: (1) an ordinal competitive goodness function that replaces contrastive pairs with competitive learning between partitioned neuron groups under distance-aware ordinal supervision; (2) a stratified ladder architecture where shallow layers learn coarse ordinal discrimination and deeper layers refine into fine-grained regression, with multi-scale feature aggregation for inter-layer collaboration; and (3) hierarchical prediction with uncertainty estimation, where multi-scale predictors jointly provide robust predictions and a single-pass uncertainty score. Extensive experimental results show FFR recovers on average 98.5% of BP's accuracy across six real-world regression benchmarks while reducing peak training memory to only 27% of BP's at depth 8 and 8% at depth 32, with per-iteration time around 72% of BP's, and substantially outperforms all BP-free competitors.

Xinyang Liu, Xuanyu Liang, Shiqi Ding +4

University of Bristol · University College London · University of Cambridge

Neural Network Optimization Feedforward Neural Networks Local Learning Forward-Forward Learning

May 29, 2026cs.NE

Dynamics and Representation Structure of Local Approximations to Gradient-Based Learning in Linear Recurrent Neural Networks

Biological and neuromorphic recurrent neural networks (RNNs) are subject to spatial and temporal locality constraints on the information that can plausibly be used during learning. A common strategy to satisfy these constraints is to modify gradient descent by neglecting non-local terms to varying degrees, as in random feedback local online (RFLO) learning and truncated backpropagation through time (tBPTT). However, the learning dynamics of these algorithms, and how they compare with BPTT, remain poorly understood. We apply dynamical systems theory to data-aligned linear RNNs -- whose dynamics can be separated into orthogonal modes -- to compare stationary solutions, stability properties, and convergence rates, finding qualitatively distinct behaviour for RFLO versus BPTT and one-step tBPTT. We further observe that the solutions learned by RFLO are restricted to low-rank perturbations of initial parameters, a result which holds beyond the data-aligned setting. Our work provides analytical insight into how locality constraints shape learning dynamics, with implications for neuroscientific models of learning and alternative optimization approaches for RNNs.

Ezekiel Williams, Alexandre Payeur, Guillaume Lajoie

Department of Mathematics and Statistics, Universit´e de Montr´eal, Canada · Mila Qu´ebec, Canada · Imperial College London, United Kingdom

Recurrent Neural Networks Linear RNNs Neural Network Training Dynamics Local Learning

May 29, 2026cs.LG

Augmented Lagrangian Predictive Coding

Predictive coding (PC) is a local-learning alternative to backpropagation (BP), training deep networks via local energy-minimization dynamics rather than a global backward pass. We introduce Augmented Lagrangian Predictive Coding (PC-ALM), which maintains PC's inference budget but aligns each weight update toward BP by accumulating per-layer constraint errors into a layer-local Lagrange multiplier. In linear PC networks, PC-ALM converges to an equilibrium with exact BP gradients distributed across the network via only layer-local updates. We analyze PC-ALM in nonlinear PC networks up to depth 128 and show that it matches BP performance across all width-depth regimes, notably in deep narrow networks where PC underperforms. PC-ALM introduces recurrent dynamics in each layer's activations. Compared to PC's heat flow on a scalar energy, PC-ALM dynamics are driven by dual ascent on the augmented Lagrangian. We observe "ballistic" credit propagation across very deep networks, with credit signals evenly distributed across layers, compared to PC's slow, diffusive credit propagation. Beyond the algorithm itself, the augmented Lagrangian framework offers a generalization of PC, and may yield insights into how distributed systems could compute and propagate BP-like credit signals through purely local dynamics.

Jeffrey Seely, Julian Gould

Sakana AI

Backpropagation Predictive Coding Augmented Lagrangian Method Neural Network Training Dynamics Local Learning

May 19, 2026cs.LG

Normative Networks for Source Separation via Local Plasticity and Dendritic Computation

Blind source separation (BSS) is a natural framework for studying how latent causes may be recovered from sensory mixtures, but deriving online and biologically plausible algorithms for structured (i.e., constrained to known domains) and potentially correlated sources remains challenging. Recent work has derived neural networks for BSS from maximization of an entropy measure, yet its online implementations involve complex and nonlocal recurrent dynamics. Motivated by this perspective, we propose Predictive Entropy Maximization, which achieves competitive performance in BSS, using only local weight updates. The method employs a close approximation of an entropy measure, yielding an objective function with easily interpretable components. Minimizing this objective leads to a predictive neural architecture in which feedforward synapses follow an error-driven rule (that can be realized through dendritic mechanisms), lateral inhibitory connections are learned with local Hebbian plasticity, and source-domain constraints are enforced through simple output nonlinearities. We derive explicit spectral bounds on the surrogate error, characterizing when the approximation is accurate. Empirically, Predictive Entropy Maximization remains robust under increasing source correlation and observation noise, outperforms biologically plausible algorithms that rely on stronger independence or decorrelation assumptions, and remains competitive with exact determinant- and correlative-information-based baselines. These results show how local plasticity and adaptive lateral inhibition can emerge from maximizing a regularized second-order entropy over structured source domains. Our implementation code is available at https://github.com/BariscanBozkurt/Predictive-Entropy-Maximization.

Bariscan Bozkurt, Efe Ali Gorguner, Francesco Innocenti +1

Gatsby Computational Neuroscience Unit, University College London, UK · Brain Network Dynamics Unit, University of Oxford, UK · Department of Computer Science, University of Oxford, UK +1

Local Learning Entropy Regularization

May 19, 2026cs.LG

Closed-form predictive coding via hierarchical Gaussian filters

Predictive coding (PC) offers a local and biologically grounded alternative to backpropagation in the training of artificial neural networks, yet to date, it remains slower, and performance degrades sharply as network depth increases. We trace both problems to a single simplification: current PC networks fix the precision matrix to the identity, discarding precision-weighted prediction errors that the variational derivation requires to be fast, local, and Bayesian. We close this gap by expressing predictive coding networks as deep hierarchical Gaussian filters (HGFs) and restore precision-weighted message passing, yielding dynamic uncertainty estimates and Hebbian-compatible update rules at every layer. The resulting networks can simultaneously learn activations, weights, and precisions under a single free-energy objective, with no global error signal, and resolve inference without requiring iterations or automatic differentiation. On FashionMNIST, our solution approaches backpropagation in epoch-level wall-clock cost while converging in fewer epochs, and outperforms it on online, data efficiency, and concept-drift tasks. We thus establish that closed-form variational inference with online precision learning provides a tractable foundation for deep predictive coding networks, retaining biological and interpretative advantages, without requiring iterative relaxation or global error signals.

Aleksandrs Baskakovs, Sylvain Estebe, Kenneth Enevoldsen +3

Center for Humanities Computing Aarhus University Aarhus, Denmark · Interacting Minds Center Aarhus University Aarhus, Denmark

Predictive Coding Variational Inference Local Learning

May 18, 2026cs.LG

Self-supervised local learning rules learn the hidden hierarchical structure of high-dimensional data

The brain learns abstract representations of high-dimensional sensory input, but the plasticity rules that enable such learning are unknown. We study biologically plausible algorithms on the Random Hierarchy Model (RHM), an artificial dataset designed to investigate how deep neural networks learn the intrinsic hierarchical structure of high-dimensional data. We focus on two types of local learning rules that avoid both a long convergence time and the use of a symmetric error network. The first type uses direct feedback signals to approximate error propagation from the output layer. The second type uses layerwise self-supervised contrastive or non-contrastive loss functions that do not explicitly approximate errors at the output layer. We show that all rules of the first type fail to solve the tasks of the RHM and trace this failure back to input-specific nonlinearities (`masking') that are implemented in full backpropagation and are essential for learning complex tasks. However, algorithms of the second type are able to learn the hierarchical hidden structure of the RHM tasks and are as data-efficient as supervised backpropagation training, while being compatible with known rules of synaptic plasticity in cortex.

Ariane Delrocq, Wu S. Zihan, Guillaume Bellec +1

School of Life Science and School of Computer and Communications Sciences, EPFL, Lausanne, Switzerland · Machine Learning Research Unit, TU Wien, Vienna, Austria

Self-Supervised Learning Hierarchical Representation Learning Biologically Plausible Learning Local Learning

May 14, 2026cs.NE

NeuroTrain: Surveying Local Learning Rules for Spiking Neural Networks with an Open Benchmarking Framework

The rapid expansion of spiking neural networks (SNNs) has led to a proliferation of training algorithms that differ widely in biological inspiration, computational structure, and hardware suitability. Despite this progress, the field lacks a unified, fine-grained taxonomy that systematically organizes these approaches and clarifies their conceptual relationships. This survey provides a comprehensive taxonomy of SNN training algorithms, spanning surrogate-gradient backpropagation, local and three-factor learning rules, biologically inspired plasticity mechanisms, ANN-to-SNN conversion pipelines, and non-standard optimization strategies. We analyze each class in terms of its computational principles, learning signals, and locality properties. To support reproducible research, we release NeuroTrain, an open-source snnTorch-based framework that implements a representative set of these algorithms within a unified, modular, and extendable framework, enabling consistent benchmarking across datasets, architectures, and training regimes. By consolidating fragmented literature and providing a reusable benchmarking framework, this survey identifies common patterns, highlights open challenges, and outlines promising directions for future work on scalable, efficient SNN training.

Alessio Caviglia, Filippo Marostica, Roberta Bardini +2

Politecnico di Torino, Control and Computer Engineering Department, Corso Duca degli Abruzzi 24, Torino, I-10129, , Italy

Spiking Neural Networks Local Learning

May 11, 2026cs.LG

Adaptive Multi-Scale Goodness Aggregation for Forward-Forward Learning

We propose Adaptive Multi-Scale Goodness Aggregation (AMSGA), a novel extension of the Forward-Forward (FF) algorithm designed to improve stability, robustness, and generalization in local-learning neural networks. AMSGA addresses several limitations of the original FF framework by introducing multi-scale goodness aggregation across local, intermediate, and global representations; adaptive curriculum-guided hard negative mining; layer-dependent adaptive thresholds; and a warm-up cosine annealing learning-rate schedule for improved optimization stability. Together, these modifications strengthen the FF paradigm while preserving its biologically plausible and memory-efficient properties. Experiments on MNIST and Fashion-MNIST demonstrate consistent performance improvements over the baseline FF algorithm, achieving up to +1.45% improvement on MNIST and +1.50% improvement on Fashion-MNIST without significant computational overhead. Our results suggest that local learning methods can become substantially more competitive when goodness estimation and training dynamics are carefully designed.

Salar Beigzad, Vansh Verma

Computer Engineering University of St. Thomas Minnesota, USA

Neural Network Optimization Local Learning Forward-Forward Learning

May 7, 2026cs.LG

Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant

Forward-Forward (FF) training lets each layer learn from a local goodness criterion. In cumulative-goodness variants, later layers can inherit a task that earlier layers have partly separated. We formalize this as layer free-riding: under the softplus FF criterion, the class-discrimination gradient reaching block

d

decays exponentially with the positive margin accumulated by earlier blocks (Theorem 3.1); a squared-hinge barrier reproduces the pathology. History-free and hardness-gated repairs raise deeper-layer separation by up to

5\times

on CIFAR-100 and

45\times

in an 8-block CIFAR-10 model, yet these and a depth-scaled auxiliary term move accuracy by under one percentage point among non-degenerate variants; on Tiny ImageNet, a harder cross-dataset check of the selected configuration, layer health likewise does not track accuracy. A theory-derived attenuation-compensated objective raises deepest-block separation

5.9\times

at a consistent cost in the FF classifier's own accuracy that a probe trained on the frozen features recovers; in a smaller CIFAR-10 control, most of its separation gain comes from dropping the depth-scaled auxiliary term it replaces. Accuracy invariance is a property of the deployed sum-based scoring rule under redistribution (Proposition 3.2) and, empirically, of the structured trained readouts we test, not of what the network learns: retrained variants disagree on 13-26% of test predictions. Architecture and augmentation matter far more than the non-degenerate training rules studied. Cumulative free-riding is therefore a real and repairable pathology, but for the softplus and squared-hinge objectives, architectures, and datasets we study, it is not the dominant factor limiting accuracy.

Amirhossein Yousefiramandi

Neural Network Training Dynamics Local Learning Forward-Forward Learning

May 6, 2026cs.CL

Rethinking Local Learning: A Cheaper and Faster Recipe for LLM Post-Training

LLM post-training typically propagates task gradients through the full depth of the model. Although this end-to-end structure is simple and general, it couples task adaptation to full-depth activation storage, long-range backward dependencies and direct task-gradient access to pretrained representations. We argue that this full-depth backward coupling can be unnecessarily expensive and intrusive, particularly when post-training supervision is much narrower than pre-training. To this end, we propose \textbf{LoPT}: Local-Learning Post-Training, a simple post-training strategy that makes gradient reach an explicit design choice. LoPT places a single gradient boundary at the transformer midpoint: the second-half block learns from the task objective, while the first-half block is updated by a lightweight feature-reconstruction objective to preserve useful representations and maintain interface compatibility. LoPT shortens the task-induced backward path while limiting direct interference from narrow task gradients on early-layer representations. Extensive experiments demonstrate that LoPT achieves competitive performance with lower memory cost, higher training efficiency and better retention of pretrained capabilities. Our code is available at: https://github.com/HumyuShi/LoPT

Hengyu Shi, Tianyang Han, Peizhe Wang +3

1Independent Researcher · 2D4 Lab · 3Southeast University

Efficient Language Model Training LLM Post-Training Local Learning

May 5, 2026cs.LG

Covariance-Aware Goodness for Scalable Forward-Forward Learning

The Forward-Forward algorithm eliminates global gradient flow and full network activations storage. However, in convolutional settings, existing BP-free FF methods significantly under-perform backpropagation on complex benchmarks such as ImageNet-100 and Tiny-ImageNet. We identify this gap as a structural bottleneck in goodness extraction: standard sum-of-squares formulation collapses feature volumes into channel-wise activation energies which omits critical second-order dependencies. To address this, we propose a framework centered on three key components. First, Bi-axis Covariance Goodness(BiCovG) explicitly augments the standard goodness function with structured second-order information along two axes: cross-channel projections that model inter-feature covariance, and nested multi-scale aggregation that encodes spatial correlation statistics. This provides a tractable approximation to covariance-aware goodness without the prohibitive O(C^2) complexity of explicit matrix estimation. Second, a lightweight Logistic Fusion module aggregates layer-wise predictions, amplifying the contribution of deeper representations. Third, the Feature Alignment Layer(FAL) introduces a zero-initialized correction at block boundaries to mitigate representation misalignment in deep locally trained networks. By introducing these three components, we effectively double the depth of viable Forward-Forward learning, extending robust layer utilization from shallow baselines to 16 layer architectures like VGG-16. The resulting BP-free model achieves 73.01% on ImageNet-100 and 50.30% on Tiny-ImageNet. As a practical extension, Hybrid Goodness Blocks control the scope of gradient propagation via configurable block sizes, further narrowing the ImageNet-100 gap to 3.6% and matching BP on Tiny-ImageNet, while still reducing peak memory by approximately 50% relative to BP.

Xiaoyi Jiang, Bashir M. Al-Hashimi, Kai Xu

King’s College London, United Kingdom · University of Nottingham

Convolutional Neural Networks Local Learning Forward-Forward Learning Image Classification

Apr 30, 2026cs.LG

Hyperspherical Forward-Forward with Prototypical Representations

The Forward-Forward (FF) algorithm presents a compelling, bio-inspired alternative to backpropagation. However, while efficient in training, it has a computationally prohibitive inference process that requires a separate forward pass for every class that is evaluated. In this work, we introduce the Hyperspherical Forward-Forward (HFF), a novel reformulation that resolves this critical bottleneck. Our core innovation is to reframe the local objective of each layer from a binary goodness-of-fit task to a direct multi-class classification problem within a hyperspherical feature space. We achieve this by learning a set of class-specific, unit-norm prototypes that act as geometric anchors and implicit negatives. This architectural innovation preserves the benefits of local training while enabling weight update and inference in a single forward pass, making it >40x faster than the original FF algorithm. Our method is simple to implement, scales effectively to modern convolutional architectures, and achieves superior accuracy on standard image classification benchmarks, closing the gap with backpropagation. Most notably, we are among the first greedy local-learning methods to report over 25% top-1 accuracy on ImageNet-1k, and 65.96% with transfer learning.

Shalini Sarode, Brian Moser, Joachim Folz +4

German Research Center for Artificial Intelligence (DFKI) · RPTU Kaiserslautern-Landau

Prototype-Based Learning Efficient Neural Network Inference Hyperspherical Representation Learning Geometric Representation Learning Local Learning Forward-Forward Learning+1

Apr 29, 2026cs.LG

Causal Learning with Neural Assemblies

Can Neural Assemblies -- groups of neurons that fire together and strengthen through co-activation -- learn the direction of causal influence between variables? While established as a computationally general substrate for classification, parsing, and planning, neural assemblies have not yet been shown to internalize causal directionality. We demonstrate that the inherent operations of neural assemblies -- projection, local plasticity control, and sparse winner selection -- are sufficient for directional learning. We introduce DIRECT (DIRectional Edge Coupling/Training), a mechanism that co-activates source and target assemblies under an adaptive gain schedule to internalize directed relations. Unlike backpropagation-based methods, DIRECT relies solely on local plasticity, making the resulting causal claims auditable at the mechanism level. Our findings are verified through a dual-readout validation strategy: (i) synaptic-strength asymmetry, measuring the emergent weight gap between forward and reverse links, and (ii) functional propagation overlap, quantifying the reliability of directional signal flow. Across multiple domains, the framework achieves perfect structural recovery under a supervised, known-structure setting. These results establish neural assemblies as an auditable bridge between biologically plausible dynamics and formal causal models, offering an "explainable by design" framework where causal claims are traceable to specific neural winners and synaptic asymmetries.

Evangelia Kopadi, Dimitris Kalles

Hellenic Open University Greece

Neural Processes Causal Discovery Causal Structure Learning Biologically Plausible Learning Causal Interpretability Local Learning

Feb 20, 2026cs.LG

Learning Long-Range Dependencies with Temporal Predictive Coding

Temporal Predictive Coding provides a layer-local, parallelisable mechanism for learning in recurrent systems, making it an attractive candidate for online local learning on neuromorphic and edge hardware. However, its recurrent parameter update captures only local temporal relationships, neglecting the historic influence of parameters along the latent-state trajectory, and therefore struggles to assign credit over longer temporal horizons. This work combines for the first time Temporal Predictive Coding with Real-Time Recurrent Learning (tPC-RTRL), incorporating an online influence matrix that tracks this historic effect whilst preserving the spatial and temporal locality properties valued by neuromorphic implementations. Under explicit assumptions, we prove that tPC-RTRL recovers the gradients of backpropagation-through-time exactly. Empirically, a near-equivalence holds across several tasks of varying scale and complexity, including byte-level language modelling on WikiText-103 (tPC-RTRL vs. BPTT: 1.865 vs. 1.864 validation BPC), English--French translation on a CCMatrix subset (20.23 vs. 20.29 BLEU), and a realistic nanodrone system-identification benchmark (0.506m vs. 0.505m mean position error). Finally, we show that the iterative inference mechanism used during training can be reused at deployment time to incorporate intermittent state observations, halving final-position error relative to open-loop rollout on the nanodrone task (0.402m vs. 0.805m) and suggesting a path towards unifying learning and filtering within the same computational framework.

Tom Potter, Oliver Rhodes

International Centre for Neuromorphic Systems, The University of Manchester, Manchester, United Kingdom

Predictive Coding Recurrent Neural Networks Local Learning

Jun 30, 2025q-bio.NC

Neural Langevin Machine: a local asymmetric learning rule can be creative

Fixed points of recurrent neural networks can be leveraged to store and generate information. These fixed points are captured by the Boltzmann-Gibbs measure, which leads to neural Langevin dynamics that relax to those fixed points for generative learning of a real dataset. We call this type of generative model a neural Langevin machine, which derives an asymmetric and firing-rate-speed-adjusted learning rule requiring only local neural signals, thereby bearing biological relevance in terms of local predictive learning. An out-of-equilibrium regime of the generative process is revealed, together with a memorization-to-generalization transition with increasing training data size. The neuro-inspired machine can also realize a continuous exploration of the phase space for different kinds of generative images and can denoise a corrupted image as well.

Zhendong Yu, Weizhong Huang, Haiping Huang

PMI Lab, School of Physics, Sun Yat-sen University, Guangzhou 510275, People’s Republic of China · Guangdong Provincial Key Laboratory of Magnetoelectric Physics and Devices, Sun Yat-sen University, Guangzhou 510275, People’s Republic of China

Neural Network Generalization Image Generation Recurrent Neural Networks Energy-Based Models Generative Modeling Local Learning