Local Learning

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 34

All topics

SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learning
Sep 28, 2026Bojian Yin, Shurong Wang, Yuqi Pan +1Language Model Pretraining Efficient Language Model Training
Institute of Automation, Chinese Academy of Sciences
Online local learning for generative thermodynamic computing
Sep 14, 2026Huilin Wang, Weibing DengThermodynamic Computing Local Learning
Key Laboratory of Quark and Lepton Physics (MOE) and Institute of Particle Physics, Central China Normal University, Wuhan 430079, China
A Study of Hidden-State Optimization Order in Predictive Coding Networks
Sep 1, 2026Xueyuan Li, Danilo Vasconcellos VargasPredictive Coding Neural Network Optimization
Conservation laws determine what physical learning remembers
Jul 30, 2026Bijaya DangolEquilibrium Propagation Neural Network Training Dynamics
1Independent researcher
The Art of Not Forgetting A Local Learning Architecture for Continual Learning
Jul 29, 2026Ashmith Atmuri, Yashaswini Rao BhogarajulaMemory-Augmented Neural Networks Continual Learning
Arkadhi Research
Reconstructing Backpropagation from Forward Fluctuations in Noise-modulated Neural Networks
Jul 29, 2026Shuhei IkemotoBackpropagation Neuromorphic Computing
Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, Kitakyushu, Japan
The Art of Not Forgetting
Jul 20, 2026Ashmith Atmuri, Akshay Kumar, Yashaswini Rao BhogarajulaMemory-Augmented Neural Networks Domain-Incremental Learning
Arkadhi Labs
Weight Feedback Computes the Jacobian Transpose Locally in Modern Deep Networks
Jul 15, 2026Junlong Shen, Xingyu LiPredictive Coding Local Learning
Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada · Alberta Machine Intelligence Institute (Amii), Edmonton, Canada
Architecture Generalization with MetaNCA
Jul 8, 2026Meet Barot, Daniel Berenberg, Sina KhajehabdollahiNeural Cellular Automata Neural Network Generalization
Mythos Scientific, New York, USA · Independent Scholar
Local Pheromone Network: Sparse Local Learning with Multi-Scale Synaptic Trails, Consolidation, and Replay
Jun 22, 2026Xingcheng Fu, Xianjun Chen, Zhihao LiHebbian Learning Continual Learning
Error Highways: Scaling Predictive Coding to Very Deep Networks
Jun 22, 2026Amirhossein Mohammadi, Alexander G. OrorbiaPredictive Coding Local Learning
SingularityNET Zug, Switzerland · Rochester Institute of Technology Rochester, NY, USA
Noise-Debiased Thermodynamic Variance for Local Learning Coefficient Probes
Jun 21, 2026Yingjia CaiSingular Learning Theory Neural Network Training Dynamics
Breaking chains with trees: Deep learning with $\mathcal{O}(\log N)$ parallel time complexity
Jun 19, 2026Neeraj Mohan Sushma, Aditya Nagarsekar, Cabrel Teguemne Fokam +4Deep Learning Optimization Neural Network Optimization
Center for Cognitive Interaction Technology, Bielefeld University, Germany · Ruhr Universität Bochum, Germany · Birla Institute of Technology and Science, Pilani, Goa Campus +1
Coercivity and Local Convergence of Physical Learning in Linear Circuits
Jun 13, 2026Joshua A. McGinnis, Xinbo Li, Yoichiro MoriEquilibrium Propagation Optimization Convergence Analysis
Department of Mathematics, University of Pennsylvania, Philadelphia.
Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training
Jun 4, 2026Yucheng ChenBackpropagation Local Learning
Amplimit Toronto, ON, Canada
FFR: Forward-Forward Learning for Regression
Jun 2, 2026Xinyang Liu, Xuanyu Liang, Shiqi Ding +4Neural Network Optimization Feedforward Neural Networks
University of Bristol · University College London · University of Cambridge
Dynamics and Representation Structure of Local Approximations to Gradient-Based Learning in Linear Recurrent Neural Networks
May 29, 2026Ezekiel Williams, Alexandre Payeur, Guillaume LajoieRecurrent Neural Networks Linear RNNs
Department of Mathematics and Statistics, Universit´e de Montr´eal, Canada · Mila Qu´ebec, Canada · Imperial College London, United Kingdom
Augmented Lagrangian Predictive Coding
May 29, 2026Jeffrey Seely, Julian GouldBackpropagation Predictive Coding
Sakana AI
Normative Networks for Source Separation via Local Plasticity and Dendritic Computation
May 19, 2026Bariscan Bozkurt, Efe Ali Gorguner, Francesco Innocenti +1Local Learning Entropy Regularization
Gatsby Computational Neuroscience Unit, University College London, UK · Brain Network Dynamics Unit, University of Oxford, UK · Department of Computer Science, University of Oxford, UK +1
Closed-form predictive coding via hierarchical Gaussian filters
May 19, 2026Aleksandrs Baskakovs, Sylvain Estebe, Kenneth Enevoldsen +3Predictive Coding Variational Inference
Center for Humanities Computing Aarhus University Aarhus, Denmark · Interacting Minds Center Aarhus University Aarhus, Denmark
Self-supervised local learning rules learn the hidden hierarchical structure of high-dimensional data
May 18, 2026Ariane Delrocq, Wu S. Zihan, Guillaume Bellec +1Self-Supervised Learning Hierarchical Representation Learning
School of Life Science and School of Computer and Communications Sciences, EPFL, Lausanne, Switzerland · Machine Learning Research Unit, TU Wien, Vienna, Austria
NeuroTrain: Surveying Local Learning Rules for Spiking Neural Networks with an Open Benchmarking Framework
May 14, 2026Alessio Caviglia, Filippo Marostica, Roberta Bardini +2Spiking Neural Networks Local Learning
Politecnico di Torino, Control and Computer Engineering Department, Corso Duca degli Abruzzi 24, Torino, I-10129, , Italy
Adaptive Multi-Scale Goodness Aggregation for Forward-Forward Learning
May 11, 2026Salar Beigzad, Vansh VermaNeural Network Optimization Local Learning
Computer Engineering University of St. Thomas Minnesota, USA
Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant
May 7, 2026Amirhossein YousefiramandiNeural Network Training Dynamics Local Learning
Rethinking Local Learning: A Cheaper and Faster Recipe for LLM Post-Training
May 6, 2026Hengyu Shi, Tianyang Han, Peizhe Wang +3Efficient Language Model Training LLM Post-Training
1Independent Researcher · 2D4 Lab · 3Southeast University
Covariance-Aware Goodness for Scalable Forward-Forward Learning
May 5, 2026Xiaoyi Jiang, Bashir M. Al-Hashimi, Kai XuConvolutional Neural Networks Local Learning
King’s College London, United Kingdom · University of Nottingham
Hyperspherical Forward-Forward with Prototypical Representations
Apr 30, 2026Shalini Sarode, Brian Moser, Joachim Folz +4Prototype-Based Learning Efficient Neural Network Inference
German Research Center for Artificial Intelligence (DFKI) · RPTU Kaiserslautern-Landau
Causal Learning with Neural Assemblies
Apr 29, 2026Evangelia Kopadi, Dimitris KallesNeural Processes Causal Discovery
Hellenic Open University Greece
Learning Long-Range Dependencies with Temporal Predictive Coding
Feb 20, 2026Tom Potter, Oliver RhodesPredictive Coding Recurrent Neural Networks
International Centre for Neuromorphic Systems, The University of Manchester, Manchester, United Kingdom
Neural Langevin Machine: a local asymmetric learning rule can be creative
Jun 30, 2025Zhendong Yu, Weizhong Huang, Haiping HuangNeural Network Generalization Image Generation
PMI Lab, School of Physics, Sun Yat-sen University, Guangzhou 510275, People’s Republic of China · Guangdong Provincial Key Laboratory of Magnetoelectric Physics and Devices, Sun Yat-sen University, Guangzhou 510275, People’s Republic of China
Stochastic Quantum Spiking Neural Networks with Quantum Memory and Local Learning
Jun 26, 2025Jiechen Chen, Bipin Rajendran, Osvaldo SimeoneQuantum Machine Learning Spiking Neural Networks
Department of Engineering, King’s College London, London, WC2R 2LS, UK · Institute for Intelligent Networked Systems, Northeastern University London, One Portsoken Street, London, E1 8PH, UK
Local Learning Rules for Out-of-Equilibrium Physical Generative Models
Jun 23, 2025Cyrill Bösch, Geoffrey Roeder, Marc Serra-Garcia +1Generative Modeling Score-Based Generative Modeling
Department of Computer Science, Princeton University, Princeton, NJ 08540, USA · AMOLF, Science Park 104, 1098 XG Amsterdam, The Netherlands
Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses
May 20, 2025Nazmus Saadat As-Saquib, A N M Nafiz Abeer, Hung-Ta Chien +3Neural Network Optimization Local Learning
Texas A&M University, College Station, TX 77843 · Sandia National Laboratories, Livermore, CA 94550
The General Theory of Localization Methods
Date pendingCongwei SongTransformer Attention Local Learning
Beijing Institute of Mathematical Sciences and Applications

Sep 28, 2026, cs.LGJ/K move · Enter open · S save

SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learning

Bojian Yin, Shurong Wang, Yuqi Pan, Guoqi Li

Institute of Automation, Chinese Academy of Sciences

Language Model Pretraining Efficient Language Model Training Local Learning

Open paper AskPDF arXiv

Large language models are trained with backpropagation, whose global gradient coordinates all layers but forces each to hold its activations and wait for the gradient to pass back through every deeper layer. Conventional local learning removes this update locking by training each module to predict the target through its own readout, but has not scaled to billion-parameter pretraining. We identify these private readouts as a key weakness, since they leave each module without information from deeper modules. We propose Shared-Output LOcal learning (SOLO), which replaces them with a shared, read-only copy of the final module's readout, the only one trained on the output of the whole network. Taken from the previous step, the copy transmits information from the final module without passing gradients between modules or reintroducing update locking. SOLO approaches backpropagation on Transformers of 340M to 2B parameters pretrained on 15B tokens, staying within one point in average zero-shot accuracy with a perplexity gap that narrows with scale. Readout ablations attribute SOLO's improvement over private readouts to sharing. Without update locking, each of p pipeline stages holds activations for O(1) micro-batches instead of O(p). The freed memory permits larger micro-batches, which reach up to 1.44x the best measured throughput of pipeline backpropagation on the same partition. To our knowledge, SOLO is the first local learning method to show such memory and throughput gains in billion-parameter language-model pretraining. Local learning thus becomes a practical alternative to backpropagation for large-scale pretraining.

Momentum

Related topics

Latest papers 34