Neural Network Initialization

Ask about this topic

Overview

Neural network initialization studies how starting weights affect the signals and gradients that flow through a model, and how they shape learning and generalization. Classical methods such as Xavier/Glorot, He/Kaiming, and orthogonal initialization aim to keep signals stable, while mean-field and random-matrix analyses characterize criticality and its limits at finite width and depth. Other approaches use data, spectra, physical operators, or pretrained models to construct task-informed starting weights. A central open tension is that an initializer’s benefits depend on architecture, task, and training procedure, and gains at the start of training do not always persist in final performance.

Latest papers

CardsList

51 papers8 papers in the last four weeks, up 60% on the four weeks before. 0.0% of all new papers.

  1. A Random-Matrix Criterion for Initializing Gated Recurrent Neural Networks

    May 11, 2026Tommaso Fioratti, Riccardo Marcaccioli, Francesco CasolaGated Recurrent UnitNeural Network Initialization

    Capital Fund Management, 75007 Paris, France · Institute of Mathematics, EPFL, 1015 Lausanne, Switzerland

  2. Criticality and Saturation in Orthogonal Neural Networks

    May 7, 2026Max Guillen, Jan E. GerkenNeural Network Initialization

    Department of Mathematical Sciences Chalmers University of Technology University of Gothenburg SE-412 96 Gothenburg, Sweden

  3. How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences

    May 6, 2026Mariia SeleznovaNeural Network InitializationRecurrent Neural Networks

    Ludwig-Maximilians-Universität München · Max Planck Institute of Quantum Optics

  4. Subcritical Signal Propagation at Initialization in Normalization-Free Transformers

    Apr 13, 2026Sergey AlekseevTransformerNeural Network Initialization

    Department of Physics and Astronomy Stony Brook University Stony Brook, NY 11794

  5. Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria

    Apr 12, 2026Nikodem TomczakDynamic Sparse TrainingSparse Neural Networks

    Thulge Labs, Singapore

  6. Initialisation Determines the Basin: Efficient Codebook Optimisation for Extreme LLM Quantization

    Apr 9, 2026Ian W. Kennedy, Nafise Sadat MoosaviLLM QuantizationNeural Network Quantization

    Department of Computer Science, University of Sheffield, Sheffield, UK

  7. Towards Initialization-dependent and Non-vacuous Generalization Bounds for Overparameterized Shallow Neural Networks

    Apr 1, 2026Yunwen Lei, Yufeng XieGeneralization BoundsOverparameterized Neural Network

    Department of Mathematics, The University of Hong Kong

  8. Schrödinger's Seed: Purr-fect Initialization for an Impurr-fect Universe

    Mar 31, 2026Mi chen, Renhao YeNeural Network InitializationML Reproducibility

    Kapteyn Astronomical Institute, University of Groningen, Postbus 800, 9700 AV Groningen, The Netherlands · School of Astronomy and Space Science, University of Chinese Academy of Sciences, 1 East Yanqi Lake Rd., Beijing 100049, P.R. China · Shanghai Astronomical Observatory, Chinese Academy of Sciences, 80 Nandan Rd., Shanghai 200030, China

  9. The Surprising Effectiveness of Noise Pretraining for Implicit Neural Representations

    Mar 30, 2026Kushal Vyas, Alper Kayabasi, Daniel Kim +3Implicit Neural RepresentationsNeural Network Initialization

    Rice University · University of California, Riverside

  10. Bit-Identical Medical Deep Learning via Structured Orthogonal Initialization

    Mar 30, 2026Yakov Pyotr ShkolnikovML ReproducibilityNeural Network Initialization

    Independent Researcher, Mountain View, CA, United States

  11. Prior-Informed Neural Network Initialization: A Spectral Approach for Function Parameterizing Architectures

    Mar 17, 2026David Orlando Salazar Torres, Diyar Altinses, Andreas SchwungNeural Network InitializationNeural Network Optimization

    Department of Automation Technology and Learning Systems, South Westphalia University of Applied Sciences, Soest, Germany

  12. The Deep-Match Framework for Event-Related Potential Detection in EEG

    Mar 11, 2026Marek Zylinski, Bartosz Tomasz Smigielski, Gerard CybulskiElectroencephalographyEEG Decoding

    Institute of Metrology and Biomedical Engineering, Faculty of Mechatronics, Warsaw University of Technology, Warsaw, Poland

  13. Does Semantic Noise Initialization Transfer from Images to Videos? A Paired Diagnostic Study

    Mar 3, 2026Yixiao Jing, Chaoyu Zhang, Zixuan Zhong +1Neural Network InitializationVideo Diffusion Models

    University of Michigan · University College London

  14. Normalisation and Initialisation Strategies for Graph Neural Networks in Blockchain Anomaly Detection

    Feb 27, 2026Dang Sy Duy, Nguyen Duy Chien, Kapil Dev +1Graph Neural NetworksNeural Network Initialization

    RMIT University, Hanoi, Vietnam · Vietnam National University, Hanoi, Vietnam

  15. A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning

    Feb 23, 2026Nicolas Anguita, Francesco Locatello, Andrew M. Saxe +4Neural Network GeneralizationRepresentation Learning

    Department of Engineering, University of Cambridge · Institute of Science and Technology, Austria (ISTA) · Gatsby Computational Neuroscience Unit and Sainsbury Wellcome Centre, UCL +1

  16. Path-conditioned training: a principled way to rescale ReLU neural networks

    Feb 23, 2026Arthur Lebeurrier, Titouan Vayer, Rémi GribonvalReLU Neural NetworksNeural Network Training Dynamics

    ENS de Lyon, CNRS, Inria, Université Claude Bernard Lyon 1, LIP, UMR 5668, 69342, Lyon cedex 07, France · Inria, Rennes, France · Inria, CNRS, ENS de Lyon, Université Claude Bernard Lyon 1, LIP, UMR 5668, 69342, Lyon cedex 07, France

  17. Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression

    Feb 19, 2026Akira Sakai, Yuma IchikawaModel CompressionNeural Network Compression

    Fujitsu Limited · Tokai University · Riken Center for AIP

  18. Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks

    Aug 30, 2025Hyungu Lee, Taehyeong Kim, Hayoung ChoiNeural Network InitializationReLU Neural Networks

    Department of Mathematics & Nonlinear Dynamics and Mathematical Application Center, Kyungpook National University, Daegu 41566, Republic of Korea · Nonlinear Dynamics and Mathematical Application Center, Kyungpook National University, Daegu 41566, Republic of Korea

  19. Optimal Condition for Initialization Variance in Deep Neural Networks: An SGD Dynamics Perspective

    Aug 18, 2025Hiroshi Horii, Sothea HasFokker–Planck EquationNeural Network Initialization

    SU · KHM

  20. Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization

    May 28, 2025Etienne Boursier, Matthew Bowditch, Matthias Englert +1Loss LandscapeOverparameterized Neural Network

    INRIA LMO, Université Paris-Saclay Orsay, France · Mathematics Institute University of Warwick Coventry, UK · Department of Computer Science University of Warwick Coventry, UK

  21. Causal pieces: analysing and improving spiking neural networks piece by piece

    Apr 18, 2025Dominik Dold, Philipp Christian PetersenNeural Network InitializationSpiking Neural Networks