Neural Network Training Dynamics

Ask about this topic

Momentum

41 papers in the last four weeks, up 273% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 349

All topics
CardsList

  1. Prospective Prediction of OOD Degradation from Source-Side Training Dynamics

    Oct 8, 2026Sasha, MoninOOD GeneralizationNeural Network Training Dynamics

    Alexander · University of South Carolina 712 Main St, 404 Columbia SC, 29208

  2. Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks

    Oct 8, 2026Yunji Wang, Junjie Yao, Linyu Liu +2Neural Network Training DynamicsFrequency-Domain Feature Learning

    School of Mathematical Sciences, Shanghai Jiao Tong University · Huawei Technologies Ltd · Shanghai University of Finance and Economics +1

  3. Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD

    Oct 8, 2026Ruoyu Zhao, Mingxuan Zhang, Jianbo Dai +3Neural Network Training DynamicsStochastic Gradient Descent

    City University of Hong Kong · Microsoft · Copula Lab +1

  4. Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals

    Oct 8, 2026Lei Zhao, Qichao Zhao, Bowen Zuo +1On-Policy DistillationLanguage Model Distillation

    University of Pennsylvania · Tsinghua University · University of California, Riverside +1

  5. Fluctuations of Nonlinear Observables in Mean Field Neural Network Training

    Oct 7, 2026Arnaud Descours, Geoffrey LacourNeural Network Training DynamicsUncertainty Quantification

    UCBL · MaIAGE

  6. When does a network's training history predict its future learning better than its current state? Evidence from a response probe and a forecasting screen

    Oct 7, 2026Martin Hofmann, Patrick MäderNeural Network Training Dynamics

    Data-intensive Systems and Visualization Group (dAI.SY), Technische Universität Ilmenau, Max-Planck-Ring 14, 98693 Ilmenau, Thuringia, Germany · German Centre for Integrative Biodiversity Research (iDiv) Halle–Jena–Leipzig, Deutscher Platz 5e, 04103 Leipzig, Saxony, Germany · Faculty of Biological Sciences, Friedrich Schiller University, Fürstengraben 1, 07745 Jena, Thuringia, Germany

  7. Stability and Diversity of Networked Self-Consuming Generative Ecosystems

    Oct 7, 2026Xiukun Wei, Yang Zhang, Xueru ZhangGenerative ModelingNeural Network Training Dynamics

    The Ohio State University

  8. Global Exponential Convergence of Two-Layer Linear Network Training

    Oct 7, 2026Stephen Y Zhang, Gabriel PeyréNeural Network Training DynamicsGradient Descent Dynamics

    Flatiron Institute & NYU · CNRS, ENS, PSL Univ.

  9. Quadratic Weak-to-Strong Generalization in Random Feature Networks via Random Matrix Theory

    Oct 6, 2026Deborah Oliveira, Elliot PaquetteWeak-to-Strong GeneralizationRandom Feature Methods

    McGill University Mila – Quebec AI Institute

  10. Neural Fields Encode Adaptation Geometry

    Oct 5, 2026Prateik Sinha, Stefania DrugaImplicit Neural RepresentationsNeural Network Training Dynamics

    Machine Learning Department Carnegie Mellon University Pittsburgh, PA 15213, USA · Sakana AI Tokyo, Japan

  11. Mind the Drift: Diagonal Linear Networks Under Large Learning Rates

    Oct 5, 2026Aniket Sanyal, Tom Jacobs, Rebekka BurkholzEdge of StabilityDeep Learning Optimization

    CISPA Helmholtz Center for Information Security · Technical University of Munich · Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA)

  12. Generalization in Neural Networks Through the Lens of Magnitude Potential

    Oct 1, 2026Sahel Torkamani, Henry Gouk, Rik SarkarNeural Network GeneralizationNeural Representation Geometry

    School of Informatics University of Edinburgh

  13. Why Does Train-Validation Separation Emerge? Update-Pressure Density Dynamics in Pretrained Backbones

    Oct 1, 2026Yuchen Li, Mingyu Du, Zongqi Fan +2Neural Network GeneralizationTransfer Learning

    School of Computer Science, The University of Sydney, Sydney, Australia · School of Computer Science and Engineering, UNSW Sydney, Sydney, Australia

  14. Increasing Width Allows Greedy Layer-wise Training to Rival End-to-End Backpropagation in Self-Supervised Learning

    Sep 30, 2026Syon Mansur, Joel ZylberbergDeep Learning OptimizationConvolutional Neural Networks

    Jules Stein Eye Institute University of California, Los Angeles Los Angeles, CA 90095, USA

  15. Disentangling Computation in Multi-Task Neural Networks with the Green's Operator

    Sep 30, 2026James HazeldenRecurrent Neural NetworksNeural Network Interpretability

    Applied Mathematics, University of Washington · The Allen Institute Seattle, WA, USA

  16. The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation

    Sep 30, 2026S. Aaron McClendon, Jorge Gallego-Feliciano, Antonios SaravanosWeight DecayAttention Sinks

    Aimpoint Digital Labs, Atlanta, GA, USA · New York University, New York, NY, USA

  17. How Does Local Landscape Geometry Evolve in Language Model Pre-Training?

    Sep 30, 2026Zhanpeng Zhou, Yuhan Sun, Bingrui Li +4Language Model PretrainingLearning Rate Scheduling

    Shanghai Jiao Tong University · Tsinghua University · Peking University

  18. Stable Transformers for Graph Generation

    Sep 30, 2026Luca Miglior, Alessio Gravina, Davide BacciuRepresentation CollapseGraph Generation

    Department of Computer Science, University of Pisa, Italy

  19. The Conflict Between Logic and Memory: Learning Higher-Order Interactions in Shallow MLPs

    Sep 30, 2026Gongyue Zhang, Honghai LiuShallow Neural NetworksMultilayer Perceptrons

  20. Awakening of the Buddha: Subspace Learning During Population-Loss Plateaus

    Sep 30, 2026Akash KumarRepresentation LearningReLU Neural Networks

    Department of Computer Science & Engineering University of California-San Diego

  21. A Dynamical Theory of LoRA in Continual Learning

    Sep 30, 2026Théo Marchetta, Filippo Alessandroni, Alessandro Breccia +2Continual LearningLow-Rank Adaptation

    Department of Mathematics, Alma Mater Studiorum – Università di Bologna, Piazza di Porta San Donato 5, 40126 Bologna, Italy · Gatsby Computational Neuroscience Unit, University College London · Donders Centre for Neuroscience, Radboud University, Nijmegen, The Netherlands

  22. Learning Under Forgetting: Statistical Support-Selective Retention in Stochastic Training Dynamics

    Sep 30, 2026Fujie Gao, Zuyue Zhang, Gang SunNeural Network MemorizationNeural Network Training Dynamics

  23. Where Does Randomness Matter in Neural Cellular Automata?

    Sep 29, 2026Fei Zuo, Jiaqi Shi, Yujing LiuNeural Cellular AutomataCellular Automata

    Fudan University · Independent Researcher · Shanghai Pupuda Culture Communication Co., Ltd.

  24. The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

    Sep 28, 2026Yihe Zhou, Tongtian Zhu, Yingxiao Huo +4Stochastic OptimizationSign-Based Optimization

    Department of Computer Science, The University of Manchester, Manchester, UK · College of Computer Science and Technology, Zhejiang University, Hangzhou, China

  25. First Learn, Then Memorize: The Spectral Bias of Diffusion Models

    Sep 28, 2026Raphaël Urfin, Tony Bonnaire, Giulio Biroli +1Neural Network GeneralizationMemorization in Generative Models

    Laboratoire de Physique de l’École normale supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université Paris Cité, F-75005 Paris, France · Université Paris-Saclay, CNRS, Institut d’Astrophysique Spatiale, 91405 Orsay, France · Department of Computing Sciences, Bocconi University, Milano, Italy

  26. Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Data

    Sep 28, 2026Jérémie Klinger, Raphaël Urfin, Giulio Biroli +1Generative ModelingScore-Based Generative Modeling

    Laboratoire de Physique de l’École normale supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université Paris Cité, F-75005 Paris, France

  27. Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons

    Sep 28, 2026Mana Sakai, Masaaki ImaizumiHyperparameter TransferNeural Network Training Dynamics

    The University of Tokyo · RIKEN Center for Advanced Intelligence Project · Kyoto University

  28. Muon Sublates the Edge of Stability in LLM Pretraining

    Sep 28, 2026Yanzhe Chen, Qifang Zhao, Xiaoxiao Xu +1Stochastic OptimizationLanguage Model Pretraining

    School of Mathematical Sciences, Shanghai Jiao Tong University, China · Alibaba Inc., Hangzhou, China · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China

  29. When Sparse Reward Meets Dense Distillation: Training Dynamics of On-Policy Distillation

    Sep 28, 2026Xinke Jiang, Tao Feng, Zhibang Yang +4Gradient InterferenceReinforcement Learning with Verifiable Rewards

    National Engineering Research Center of Software Engineering, Peking University, Beijing, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China +1

  30. Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks

    Sep 28, 2026Etienne Boursier, Nicolas FlammarionRepresentation LearningFine-Tuning

    INRIA, LMO, Universit´e Paris-Saclay, Orsay, France · EPFL, Lausanne, Switzerland