Neural Network Training Dynamics

Ask about this topic

Momentum

41 papers in the last four weeks, up 273% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 349

All topics
CardsList

  1. Understanding Two-Layer Neural Networks with Smooth Activation Functions

    Jul 11, 2025Changcun HuangNeural Network Activation FunctionsNeural Network Approximation Theory

    Shuitu Institute of Applied Mathematics, Chongqing 400700, P.R.C

  2. A statistical physics framework for optimal learning

    Jul 10, 2025Francesca Mignacco, Francesco MoriStatistical Physics of LearningNeural Network Training Dynamics

    1Joseph Henry Laboratories of Physics, Princeton University, Princeton, NJ 08544, USA · 2Graduate Center, City University of New York, New York, NY 10016, USA · Center of Mathematical Sciences and Applications, Harvard University, Cambridge, MA 02138, USA

  3. Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models

    Jun 16, 2025Zhenyu Liao, Michael W. MahoneyRandom Matrix TheoryNeural Network Generalization

    School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, China · ICSI, LBNL, and Department of Statistics, University of California, Berkeley, USA

  4. KITINet: KInetic Theory Inspired Inter-Channel Information Exchange for Residual Networks

    May 23, 2025Mingquan Feng, Yifan Fu, Tongcheng Zhang +3Feature Interaction ModelingResidual Learning

    Shanghai Jiao Tong University

  5. KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approaches

    May 20, 2025Mingquan Feng, Yixin Huang, Yifan Fu +2Neural Network OptimizationNeural Network Training Dynamics

    Shanghai Jiao Tong University

  6. Silent Neuron Theory and Plasticity Preservation for Deep Reinforcement Learning in Adaptive Video Streaming

    May 2, 2025Zhiqiang He, Zhi LiuNon-Stationary RLNeural Network Training Dynamics

    Department of Computer and Network Engineering, the University of Electro-Communications, Japan.

  7. The Dynamics of Generalization in Deep Learning

    Apr 23, 2025Rubing Yang, Pratik ChaudhariNeural Network GeneralizationNeural Network Training Dynamics

    Department of Applied Mathematics and Computational Sciences, University of Pennsylvania Philadelphia, PA, 19104, USA · Department of Electrical and Systems Engineering, University of Pennsylvania Philadelphia, PA, 19104, USA

  8. Hyperflux: Pruning Reveals Importance

    Apr 6, 2025Eugen Barbulescu, Antonio Alexoaie, Lucian BusoniuSparse Neural NetworksNeural Network Training Dynamics

    Department of Computer Science Technical University of Cluj-Napoca · Department of Automation Technical University of Cluj-Napoca

  9. AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery

    Apr 2, 2025Behnam Gheshlaghi, Shahin AtakishiyevShallow Neural NetworksRepresentation Learning

  10. Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning

    Jan 13, 2025Thomas ChenNeural Network InterpretabilityNeural Network Training Dynamics

  11. Improving the adaptive and continuous learning capabilities of artificial neural networks: Lessons from multi-neuromodulatory dynamics

    Jan 12, 2025Jie Mei, Alejandro Rodriguez-Garcia, Daigo Takeuchi +4Continual LearningNeural Network Training Dynamics

    IT:U Interdisciplinary Transformation University Austria, Linz, Austria · International Research Center for Neurointelligence, The University of Tokyo, Tokyo, Japan · Department of Anatomy, University of Quebec in Trois-Rivieres, Trois-Rivieres, QC, Canada +8

  12. Network Dynamics-Based Framework for Understanding Deep Neural Networks

    Jan 5, 2025Yuchen Lin, Yong Zhang, Sihan Feng +1Neural Network GeneralizationNeural Network Training Dynamics

    Department of Physics, Xiamen University, Xiamen 361005, China · 2Lanzhou Center for Theoretical Physics, Lanzhou University, Lanzhou 730000, China · 3Shenzhen Angel FOF Management Co., Ltd., Shenzhen 518000, China

  13. A Mechanistic Study of Transformers Training Dynamics

    Oct 31, 2024Ambroise Odonnat, Wassim Bouaziz, Vivien CabannesTransformer InterpretabilityAttention Mechanisms

    Inria, Univ. Rennes 2 · Mistral AI; work done while at Meta. · FAIR at Meta.

  14. How Learning Dynamics Drive Adversarially Robust Generalization?

    Oct 10, 2024Yuelin Xu, Xiao ZhangAdversarial TrainingPAC-Bayesian Generalization Bounds

    CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

  15. A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks

    Oct 28, 2022Zhengdao Chen, Eric Vanden-Eijnden, Joan BrunaNeural Network GeneralizationNeural Network Optimization

    Google Research Mountain View, CA 94043 · Courant Institute, New York University New York, NY 10003

  16. Deep Residual Networks Learn the Geodesic Curve in the Wasserstein Space

    Feb 18, 2021Kuo Gai, Shihua ZhangResidual LearningNeural Network Training Dynamics

    Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS) Shanghai 200433, China · Academy of Mathematics and Systems Science Chinese Academy of Sciences Beijing 100190, China · School of Mathematical Sciences2026 University of Chinese Academy of Sciences Beijing 100049, China

  17. Theory of the Frequency Principle for General Deep Neural Networks

    Jun 21, 2019Tao Luo, Zheng Ma, Zhi-Qin John Xu +1Frequency-Domain Feature LearningNeural Network Training Dynamics

    Department of Mathematics, Zheng Ma Purdue University · Department of Mathematics, Purdue University · New York University Abu Dhabi and Courant Institute

  18. ExpTest: Loss-Curve Hypothesis Testing for Autonomous Learning-Rate Selection in Deep Neural Networks

    Date pendingZan Chaudhry, Naoko MizunoLearning Rate SchedulingNeural Network Training Dynamics

    Laboratory of Structural Cell Biology, National Institutes of Health, Bethesda, MD, 20892 USA · Ragon Institute, Cambridge, MA, 02139

  19. On the Residual Scaling of Looped Transformers: Stability and Transferability

    Date pendingShaowen Wang, Bingrui Li, Ge Zhang +3Residual LearningHyperparameter Transfer

    Tsinghua University · ByteDance Seed · M-A-P