Neural Network Initialization

Ask about this topic

Overview

Neural network initialization studies how starting weights affect the signals and gradients that flow through a model, and how they shape learning and generalization. Classical methods such as Xavier/Glorot, He/Kaiming, and orthogonal initialization aim to keep signals stable, while mean-field and random-matrix analyses characterize criticality and its limits at finite width and depth. Other approaches use data, spectra, physical operators, or pretrained models to construct task-informed starting weights. A central open tension is that an initializer’s benefits depend on architecture, task, and training procedure, and gains at the start of training do not always persist in final performance.

Latest papers

CardsList

51 papers8 papers in the last four weeks, up 60% on the four weeks before. 0.0% of all new papers.

  1. Gate Initialisation versus Recurrent Dropout and Zoneout in Learning Long Dependencies: A Controlled Study of LSTM and GRU Networks on Synthetic Tasks

    Oct 21, 2026Oleksandr Kholodniak, Oleksandr ProkhorovRecurrent Neural NetworksLong Short-Term Memory Networks

    National Aerospace University “Kharkiv Aviation Institute”, Kharkiv, Ukraine

  2. Do We Need Learned Targets? Classical Gradients as Supervision for 2D Gaussian Initialization

    Oct 19, 2026Kyungjune BaekNeural Network InitializationImage Reconstruction

    Sejong University, Seoul, Republic of Korea

  3. Trainable at Depth, Solvable at Small Size: What the Loss Surface of a Polynomial Activation Dictates for Designing, Initializing and Training Networks

    Oct 14, 2026Xiuyi YangNeural Network OptimizationNeural Network Training Dynamics

  4. Ideal Paths for Approximating Logistic Gradient Descent Trajectories at Large Initialization

    Oct 2, 2026Junjie Xiao, Huiwen JiaLogistic RegressionNeural Network Initialization

    Department of Mathematics, Peking University · Department of Industrial Engineering and Operations Research, University of California, Berkeley

  5. Operator-informed initialization for Fourier features physics-informed neural networks

    Oct 2, 2026Juan Molina, Paris Perdikaris, Mircea Petrache +2Physics-Informed MLNeural Network Initialization

    National Center for Artificial Intelligence, Santiago, Chile · Department of Mechanical Engineering and Applied Mechanics, University of Pennsylvania, Philadelphia, PA, USA · Department of Mathematics, Pontificia Universidad Católica de Chile, Santiago, Chile +3

  6. Dimension-Free Rank Lifting from Random Hyperplane Arrangements

    Sep 30, 2026Luca Becchetti, Matteo Russo, Ruben SkorupinskiNeural Network Initialization

    Sapienza University of Rome, Italy · EPFL, Switzerland

  7. Procedural Core: A Compact Recurrent Initialization for Vision Transformers

    Sep 29, 2026Zachary Shinnick, Christian Internò, Hemanth Saratchandran +2Neural Network InitializationVision Transformer

    Australian Institute for Machine Learning (AIML), Adelaide University · Bielefeld University · Metacognition AI +1

  8. Copy the Same, Distill the Difference: Initializing Linear Vision Transformers

    Sep 28, 2026Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +9Neural Network InitializationVision Transformer

    Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · Nanyang Technological University · ST Engineering Geo-Insights, Singapore +3

  9. Stable initialization without the CLT

    Sep 24, 2026Simon Kuang, Kyle Chickering, Xinfan LinNeural Network InitializationNeural Network Stability

    Department of Mechanical and Aerospace Engineering University of California, Davis

  10. Structural priors for data-efficient language learning

    Sep 10, 2026Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager +1Language Model PretrainingNeural Network Initialization

    University of Göttingen, Germany

  11. Critical initialization destabilizes higher input derivatives in wide scalar-input networks

    Sep 8, 2026Prashant Singh, Pranav SinghNeural Network Initialization

    Department of Mathematics, Indian Institute of Technology Ropar, Rupnagar, Punjab, 140001, India

  12. Conditioned Initialization for Attention

    Sep 7, 2026Hemanth Saratchandran, Simon LuceyNeural Network InitializationTransformer Attention

    Australian Institute for Machine Learning Adelaide University

  13. Correlated initialization of deep residual networks

    Sep 3, 2026Felix Benning, Ivan Nourdin, Giovanni PeccatiNeural Network InitializationResidual Learning

  14. Joint Initialization of Flux Networks and Effective Multiplication Factor for Physics-Informed Neural Networks Solving Neutron Diffusion Problems

    Aug 26, 2026Qin Hang, Yangdi Yi, Jiayi Li +2Neural PDE SolversPhysics-Informed ML

    School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China · Center for Scientific Intelligence Innovation, University of Science and Technology of China, Hefei 230026, China · Institute of Advanced Technology, University of Science and Technology of China, Hefei 230088, China

  15. ScoutNeRV: Rapid Encoding of Grid-Based Video INRs via ScoutNet

    Aug 25, 2026Naser Alizada, Farhang Baghban, Hashem Pishkar +1Neural Video CompressionImplicit Neural Representations

    Department of Computer Engineering, Ne. C., Islamic Azad University, Neyshabur, Iran.

  16. Unveiling the Secret of AdaLN-Zero in Diffusion Transformer

    Aug 10, 2026Jie Zhu, Mingyu Ding, Boqiang Duan +2Diffusion TransformerNeural Network Initialization

    Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education, China, and School of Computer Science, Peking University, China. · UC Berkeley. · Baidu, Beijing, China.

  17. Correlation flow governs learning at criticality

    Aug 8, 2026Andrea Combette, Nelly Pustelnik, Antoine VenailleNeural Network InitializationNeural Network Training Dynamics

    CNRS, ENS de Lyon, Laboratoire de Physique UMR5672, Lyon, France

  18. LLM-Guided Initialization for Accelerated Hybrid Quantum-Classical Medical Image Classification

    Jul 29, 2026Riza Alaudin Syah, Irwan Alnarus Kautsar, Haza Nuzly Bin Abdull HamedNeural Network InitializationHybrid Quantum-Classical ML

    Faculty of Computing Universiti Teknologi Malaysia Johor Bahru, Malaysia · Remote Student based in Jakarta, Indonesia · Informatics Department Faculty of Science and Technology Universitas Muhammadiyah Sidoarjo Sidoarjo, Indonesia

  19. Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning

    Jul 27, 2026Luc McCutcheon, Evangelos Chatzaroulas, Saber FallahNon-Stationary RLNeural Network Initialization

    Department of Mechanical Engineering, University of Surrey, United Kingdom · Department of Computing Science, University of Surrey, United Kingdom

  20. Quasi-Monte Carlo Initialization for Meta-Reinforcement Learning

    Jul 21, 2026Julian G. SoltesNeural Network InitializationMeta-Reinforcement Learning

    Regis University, USA

  21. Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

    Jul 15, 2026Katie EverettTransformer FFNsTransformer

    MIT CSAIL

  22. Complexity-Guided Component-wise Initialization for Language Model Pretraining

    Jul 10, 2026Konstantin Garbers, Nicholas OhLanguage Model PretrainingNeural Network Initialization

    Peking University Beijing, China

  23. From Dataset Spectral Geometry to Network Weights: A Geometry-Aware Initialization for Sigmoidal MLPs in Image Classification

    Jun 26, 2026Yi-Shan ChuMultilayer PerceptronsNeural Network Initialization

    Institute of Statistical Science, Academia Sinica, Taipei, Taiwan (R.O.C.)

  24. Pre-Warm: Initializing Convolutional Filters from First-Batch Patch Dictionaries

    Jun 24, 2026Rowan MartnishnNeural Network InitializationConvolutional Neural Networks

    University of Pennsylvania

  25. Small Initialization Matters for Large Language Models

    Jun 16, 2026Liangkai Hang, Junjie Yao, Zhiyu Li +3Language Model PretrainingNeural Network Initialization

    School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China. · MemTensor (Shanghai) Technology Co., Ltd.. for Advanced Algorithms Research, Shanghai, China. · Institute of Natural Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China.

  26. Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation

    Jun 15, 2026Zhongzhu Zhou, Qingyang Wu, Junxiong Wang +4Neural Network InitializationLinear Attention

    1The University of Sydney · 2Together AI · 5Microsoft +2

  27. Robust Neural Tucker Factorization with Bias Correction and Adaptive Initialization

    Jun 15, 2026Yuchao Su, Yixin RanTensor CompletionNeural Network Initialization

    School of Computer Science and Engineering, Chongqing University of Science and Technology, Chongqing, China · College of Computer and Information Science, School of Software, Southwest University, Chongqing, China

  28. Jacobi-Anger Method for Deterministic Initialization in Implicit Neural Representation

    Jun 4, 2026Mohammed Alsakabi, Kejia Hu, John M. Dolan +1Implicit Neural RepresentationsNeural Network Initialization

    Department of Electrical and Computer Engineering, College of Engineering · The Robotics Institute, School of Computer Science Carnegie Mellon University, Pittsburgh, PA, USA

  29. Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias

    May 27, 2026Mohua Das, Pierfrancesco Beneventano, Shibshankar Dey +2Neural Network GeneralizationNeural Network Initialization

    MIT · Northwestern University

  30. CAWI: Copula-Aligned Weight Initialization for Randomized Neural Networks

    May 12, 2026Mushir Akhtar, M. Tanveer, Mohd. ArshadNeural Network Initialization

    Indian Institute of Technology Indore, Simrol, Indore, 453552, India