Neural Network Training Dynamics

Follow this topic Ask about this topic

Momentum

41 papers in the last four weeks, up 273% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 349

All topics

Advancing Direct Training for Spiking Neural Networks with Circulate-Firing Neurons and Learnable Gradients
May 14, 2026Feifan Zhou, Xiang Wei, Yang Liu +1Spiking Neural Networks Neural Network Training Dynamics
School of Artificial and Intelligence, Tianjin University, Tianjin 300072, China
Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning
May 14, 2026Shin So, Kyelim Lee, Albert NoNeural Network Generalization Neural Network Training Dynamics
Yonsei University
Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology
May 14, 2026Jesseba Fernando, Grigori GuitchountsDynamical Systems Transformer
Network Science Institute, Northeastern University · Flagship Pioneering
Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning
May 13, 2026Yatin Dandi, Matteo Vilucchio, Luca Arnaboldi +2Hierarchical Representation Learning Neural Network Training Dynamics
Information Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)
First-Passage Prediction of Grokking Delay: ACalibrated Law under AdamW with Causal Validation
May 13, 2026Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung +1Neural Network Memorization Neural Network Training Dynamics
H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam
State-Space NTK Collapse Near Bifurcations
May 12, 2026James Hazelden, Eric Shea-BrownRecurrent Neural Networks Neural Network Training Dynamics
University of Washington
Detecting overfitting in Neural Networks during long-horizon grokking using Random Matrix Theory
May 12, 2026Hari K. Prakash, Charles H MartinNeural Network Generalization Neural Network Training Dynamics
University of California San Diego Data Science and Engineering · Calculation Consulting San Francisco, CA, USA
Understanding Sample Efficiency in Predictive Coding
May 12, 2026Gaspard Oliviers, Elene Lominadze, Rafal BogaczBackpropagation Predictive Coding
Nuffield Department of Clinical Neurosciences, University of Oxford, United Kingdom · MRC Centre of Research Excellence in Restorative Neural Dynamics, United Kingdom
OUI as a Structural Observable: Towards an Activation-Centric View of Neural Network Training
May 12, 2026Alberto Fernández-Hernández, Jose I. Mestre, Cristian Pérez-Corral +3Neural Network Generalization Neural Network Training Dynamics
Universitat Politècnica de València, Valencia, Spain · Universitat Jaume I, Castelló de la Plana, Spain · Openchip & Software Technologies S.L., Spain
Error whitening: Why Gauss-Newton outperforms Newton
May 11, 2026Maricela Best McKay, Nathan P. Lawrence, Brian Wetton +1Gauss-Newton Optimization Deep Learning Optimization
University of British Columbia · University of California, Berkeley
Uniform Scaling Limits in AdamW-Trained Transformers
May 11, 2026William Gibson, Christoph ReisingerTransformer Self-Attention
Mathematical Institute · University of Oxford · Oxford, OX2 6GG, UK
On the global convergence of gradient descent for wide shallow models with bounded nonlinearities
May 11, 2026Romain Petit, Clarice Poon, Gabriel PeyréShallow Neural Networks Gradient Descent
CNRS and ENS, PSL Universit´e · Mathematics Institute, University of Warwick
Simply Stabilizing the Loop via Fully Looped Transformer
May 11, 2026Rao Fu, Zixuan Yang, Jiankun Zhang +4Test-Time Scaling Recurrent Transformers
Hong Kong Baptist University · Jilin University
The two clocks and the innovation window: When and how generative models learn rules
May 11, 2026Binxu Wang, Emma Lucia Byrnes Finn, Bingbin LiuMemorization in Generative Models Generative Modeling
Kempner Institute at Harvard University
Flag Varieties: A Geometric Framework for Deep Network Alignment
May 11, 2026Jingchuan Xiao, Xinyi Sui, Cihan RuanNeural Representation Geometry Neural Collapse
Department of Mathematics and Computer Studies, Mary Immaculate College, Ireland · Department of Computer Science and Engineering, Santa Clara University, USA
Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective
May 9, 2026Jiuqi Wang, Jayanth Srinivasa, Claire Chen +3Deep Learning Optimization Loss of Plasticity
University of Virginia · Cisco Research · California Institute of Technology +1
Spherical Boltzmann machines: a solvable theory of learning and generation in energy-based models
May 9, 2026Thomas Tulinski, Simona Cocco, Rémi Monasson +1Statistical Physics of Learning Phase Transitions
Laboratoire de Physique de l’École Normale Supérieure, PSL, CNRS UMR8023, Sorbonne Université, 24 rue Lhomond, 75005 Paris, France · Institut de Physique Théorique, Université Paris-Saclay,2026 CNRS UMR3681, CEA, Gif-sur-Yvette, France
The Propagation Field: A Geometric Substrate Theory of Deep Learning
May 8, 2026Xingrui GuNeural Network Generalization Neural Representation Geometry
1EECS, University of California, Berkeley, Berkeley, CA.
Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit
May 8, 2026Konstantin Riedl, Konstantinos Spiliopoulos, Justin SirignanoNeural Network Optimization Spectral Bias
Mathematical Institute University of Oxford · Department of Mathematics & Statistics Boston University
Convergent Stochastic Training of Attention and Understanding LoRA
May 8, 2026Zhengkai Sun, Dibyakanti Kumar, Alejandro F Frangi +2Stochastic Optimization Low-Rank Adaptation
Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer
May 8, 2026Clarissa Lauditi, Cengiz Pehlevan, Blake BordelonDeep Linear Networks Neural Network Training Dynamics
John A. Paulson School of Engineering and Applied Sciences, Harvard University · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Center for Mathematical Sciences and Applications, Harvard University +1
Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers
May 8, 2026Noboru Isobe, Daisuke Inoue, Masaaki ImaizumiTransformer Clustering
1RIKEN Center for Advanced Intelligence Project · 2Imperial College London · 3The University of Tokyo +1
Bilevel Graph Structure Learning, Revisited: Inner-Channel Origins of the Reported Gain
May 8, 2026Minkyoung Kim, Beakcheol JangGraph Structure Learning Graph Neural Networks
Graduate School of Information, Yonsei University
On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR
May 7, 2026Hao Ye, Jisheng Dang, Junfeng Fang +7Reinforcement Learning Reinforcement Learning with Verifiable Rewards
Lanzhou University · National University of Singapore · Beijing Institute of Technology
MinMax Recurrent Neural Cascades
May 7, 2026Alessandro RoncaRecurrent Neural Networks Neural Network Training Dynamics
IRIS-AI
Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networks
May 7, 2026Yi Wei, Xuan Qi, Furao ShenNeural Network Activation Functions Neural Network Approximation Theory
DITEN University of Genoa, Genoa, Italy · State Key Laboratory of Novel Software Technology School of Artificial Intelligence Nanjing University, Jiangsu, China
The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks
May 7, 2026Taehun Cha, Daniel Beaglehole, Adityanarayanan Radhakrishnan +1Neural Network Training Dynamics Gradient Descent Dynamics
Department of Mathematics Korea University · Computer Science and Engineering UC San Diego · MIT Mathematics Broad Institute of MIT and Harvard
Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant
May 7, 2026Amirhossein YousefiramandiNeural Network Training Dynamics Local Learning
Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes
May 7, 2026Hanqing Liu, Jianjun Cao, Yuanze Li +1Neural Network Training Dynamics
Tsinghua University · The University of Tokyo
Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training
May 7, 2026Hang Chen, Jiaying Zhu, Hongyang Chen +3Supervised Fine-Tuning Transformer Interpretability
School of Computer Science and Technology Xi’an Jiaotong University · School of Computer Science and Engineering The Chinese University of Hong Kong · Shaanxi Co., Ltd(Xi’an 710077, China) China Mobile Group +1

May 14, 2026, cs.NEJ/K move · Enter open · S save

Advancing Direct Training for Spiking Neural Networks with Circulate-Firing Neurons and Learnable Gradients

Feifan Zhou, Xiang Wei, Yang Liu, Qiang Yu

School of Artificial and Intelligence, Tianjin University, Tianjin 300072, China

Spiking Neural Networks Neural Network Training Dynamics

Open paper AskPDF arXiv

Spiking Neural Networks (SNNs) have emerged with promising energy-efficient property, yet a substantial performance gap persists compared to Artificial Neural Networks (ANNs). This gap stems from at least two key limitations: first, conventional spiking neurons offer limited information representation capacity, underutilizing the rich dynamics of membrane potentials; second, fixed surrogate gradient (SG) functions across time steps leads to imprecise gradient propagation, impeding effective direct training. To address these two challenges, we propose a new direct training algorithm with three core innovations: first, a circulate-firing spiking neuron model that enhances information representation capacity by leveraging membrane potentials more effectively; second, a time-step-wise learnable surrogate gradient function, enabling accurate gradient estimation during backpropagation; third, a positive-negative balanced loss function to achieve equilibrium between positive and negative membrane potentials and further boost SNN performance. Extensive experiments demonstrate that our methods achieve competitive performance across multiple datasets. Our methods can generalize seamlessly to advanced architectures of Transformer, consistently outperforming existing methods. Our work highlights the effectiveness of further harnessing intrinsic membrane dynamics of SNNs for performance improvement, and thus open a new avenue for advancing high-performance spiking neural architectures.

Momentum

Related topics

Latest papers 349