New Initialization

New initialization techniques for neural networks aim to improve training efficiency, stability, and generalization performance by carefully selecting initial model parameters. Current research focuses on developing methods tailored to specific architectures like transformers and diffusion models, often leveraging techniques such as reparameterization, knowledge factorization, and adaptive segmentation to optimize initialization for various tasks, including image generation, natural language processing, and visual navigation. These advancements are significant because they can lead to faster training, reduced computational costs, and improved model accuracy across a wide range of applications.

Papers

February 1, 2022

Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization
Mariia Seleznova, Gitta Kutyniok
Mixed Effect Neural Tangent Kernel Large Depth ReLU Network New Initialization Infinite Width Infinite Depth and Width Limit

January 28, 2022

On feedforward control using physics-guided neural networks: Training cost regularization and optimized initialization
Max Bolderman, Mircea Lazar, Hans Butler
New Initialization Feedforward Control Inverse Dynamic Model

January 27, 2022

Eigenvalues of Autoencoders in Training and at Initialization
Benjamin Dees, Susama Agarwala, Corey Lowman
Training Data Supervised Autoencoder New Initialization MNIST Dataset Random Matrix Smallest Eigenvalue Eigenvalue Distribution

January 5, 2022

Multi-layer VI-GNSS Global Positioning Framework with Numerical Solution aided MAP Initialization
Bing Han, Zhongyang Xiao, Shuai Huang, Tao Zhang
Visual Odometry Global Navigation Satellite System Global Localization New Initialization Numerical Solution

November 23, 2021

Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications
Darshil Doshi, Tianyu He, Andrey Gromov
Deep Neural Network Financial Application New Initialization Residual Connection Jacobian Matrix Criticality Ordered Spin Sequence Gaussian Approximation

New Initialization

Papers

Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization

On feedforward control using physics-guided neural networks: Training cost regularization and optimized initialization

Eigenvalues of Autoencoders in Training and at Initialization

Multi-layer VI-GNSS Global Positioning Framework with Numerical Solution aided MAP Initialization

Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications