Neural Network Initialization

Overview

Neural network initialization studies how starting weights affect the signals and gradients that flow through a model, and how they shape learning and generalization. Classical methods such as Xavier/Glorot, He/Kaiming, and orthogonal initialization aim to keep signals stable, while mean-field and random-matrix analyses characterize criticality and its limits at finite width and depth. Other approaches use data, spectra, physical operators, or pretrained models to construct task-informed starting weights. A central open tension is that an initializer’s benefits depend on architecture, task, and training procedure, and gains at the start of training do not always persist in final performance.

Latest papers

CardsList

56 papers5 papers in the last four weeks, down 29% on the four weeks before. 0.0% of all new papers.

  1. Warm Starting Neural Dynamics Models From Few Jacobian Samples

    Nov 16, 2026Maxime Rochkoulets, Joppe De Jonghe, Mariya IshtevaModel-Based RLSample-Efficient RL

  2. Rethinking How to Leverage Cheap Physics for Data-Efficient Neural Operators

    Nov 6, 2026Shuo Zhang, Senwei LiangPDE Operator LearningOperator Learning

  3. Finite-width shift of the edge of chaos in deep neural networks

    Nov 4, 2026Arn CaesarFinite-Width Neural Network TheoryNeural Network Initialization

  4. TreePIN2^{2}: Tree-Prior Initialized Neural Networks

    Oct 31, 2026Yanrong Wang, Qiming Sun, Haoyang ChenNeural Network Initialization

  5. Ideal Paths for Approximating Logistic Gradient Descent Trajectories at Large Initialization

    Oct 2, 2026Junjie Xiao, Huiwen JiaLogistic RegressionNeural Network Initialization

  6. Operator-informed initialization for Fourier features physics-informed neural networks

    Oct 2, 2026Juan Molina, Paris Perdikaris, Mircea Petrache +2Physics-Informed MLNeural Network Initialization

  7. Dimension-Free Rank Lifting from Random Hyperplane Arrangements

    Sep 30, 2026Luca Becchetti, Matteo Russo, Ruben SkorupinskiNeural Network Initialization

  8. Procedural Core: A Compact Recurrent Initialization for Vision Transformers

    Sep 29, 2026Zachary Shinnick, Christian Internò, Hemanth Saratchandran +2Neural Network InitializationVision Transformer

  9. Copy the Same, Distill the Difference: Initializing Linear Vision Transformers

    Sep 28, 2026Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +9Neural Network InitializationVision Transformer

  10. Stable initialization without the CLT

    Sep 24, 2026Simon Kuang, Kyle Chickering, Xinfan LinNeural Network InitializationNeural Network Stability

  11. Structural priors for data-efficient language learning

    Sep 10, 2026Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager +1Language Model PretrainingNeural Network Initialization

  12. Critical initialization destabilizes higher input derivatives in wide scalar-input networks

    Sep 8, 2026Prashant Singh, Pranav SinghNeural Network Initialization

  13. Conditioned Initialization for Attention

    Sep 7, 2026Hemanth Saratchandran, Simon LuceyNeural Network InitializationTransformer Attention

  14. Joint Initialization of Flux Networks and Effective Multiplication Factor for Physics-Informed Neural Networks Solving Neutron Diffusion Problems

    Aug 26, 2026Qin Hang, Yangdi Yi, Jiayi Li +2Neural PDE SolversPhysics-Informed ML

  15. ScoutNeRV: Rapid Encoding of Grid-Based Video INRs via ScoutNet

    Aug 25, 2026Naser Alizada, Farhang Baghban, Hashem Pishkar +1Neural Video CompressionImplicit Neural Representations

  16. Unveiling the Secret of AdaLN-Zero in Diffusion Transformer

    Aug 10, 2026Jie Zhu, Mingyu Ding, Boqiang Duan +2Diffusion TransformerNeural Network Initialization

  17. Correlation flow governs learning at criticality

    Aug 8, 2026Andrea Combette, Nelly Pustelnik, Antoine VenailleNeural Network InitializationNeural Network Training Dynamics

  18. LLM-Guided Initialization for Accelerated Hybrid Quantum-Classical Medical Image Classification

    Jul 29, 2026Riza Alaudin Syah, Irwan Alnarus Kautsar, Haza Nuzly Bin Abdull HamedNeural Network InitializationHybrid Quantum-Classical ML

  19. Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning

    Jul 27, 2026Luc McCutcheon, Evangelos Chatzaroulas, Saber FallahNon-Stationary RLNeural Network Initialization

  20. Small Initialization Matters for Large Language Models

    Jun 16, 2026Liangkai Hang, Junjie Yao, Zhiyu Li +3Language Model PretrainingNeural Network Initialization

  21. Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation

    Jun 15, 2026Zhongzhu Zhou, Qingyang Wu, Junxiong Wang +4Neural Network InitializationLinear Attention