New Initialization

New initialization techniques for neural networks aim to improve training efficiency, stability, and generalization performance by carefully selecting initial model parameters. Current research focuses on developing methods tailored to specific architectures like transformers and diffusion models, often leveraging techniques such as reparameterization, knowledge factorization, and adaptive segmentation to optimize initialization for various tasks, including image generation, natural language processing, and visual navigation. These advancements are significant because they can lead to faster training, reduced computational costs, and improved model accuracy across a wide range of applications.

Papers

May 27, 2023

Pruning at Initialization -- A Sketching Perspective
Noga Bar, Raja Giryes
Practical Algorithm Edge Pruning Sparse Network New Initialization Lottery Ticket Hypothesis Sparse Mask Data Independent Perspective Sketch

May 23, 2023

FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models
Konstantin Dobler, Gerard de Melo
Language Model Multilingual Model High Resource Language Human Driving Focus New Initialization Monolingual Text Sparse Softmax

April 25, 2023

BO-ICP: Initialization of Iterative Closest Point Based on Bayesian Optimization
Harel Biggie, Andrew Beathard, Christoffer Heckman
Point Cloud Bayesian Optimization Point Cloud Registration New Initialization Iterative Closest Point

April 6, 2023

Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training
Luís Carvalho, João Lopes Costa, José Mourão, Gonçalo Oliveira
Training Data Gaussian Process New Initialization Wide Neural Network Random Field Width Network Layer NTK

April 4, 2023

Effective Theory of Transformers at Initialization
Emily Dinan, Sho Yaida, Susan Zhang
Transformer Megatron Decepticons New Initialization Deep Transformer Backward Propagation Residual Neural Network Effective Theory

March 31, 2023

On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networks
Sebastian Neumayer, Lénaïc Chizat, Michael Unser
Gradient Descent Mixed Effect Convex Set Two Layer Neural Network Root to Leaf Path New Initialization Two Layer ReLU Unbalanced Optimal Transport Regularization Path

March 26, 2023

Does "Deep Learning on a Data Diet" reproduce? Overall yes, but GraNd at Initialization does not
Andreas Kirsch
Deep Learning Gradient Norm New Initialization Data Pruning Random Pruning Data Diet

February 20, 2023

Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initialization
Cameron Jakub, Mihai Nica
Neural Network Deep Neural Network ReLU Network New Initialization Tilting Angle Combinatorial Approach Model Degeneracy ReLU Function

February 8, 2023

Unsupervised Learning of Initialization in Deep Neural Networks via Maximum Mean Discrepancy
Cheolhyoung Lee, Kyunghyun Cho
Deep Learning Deep Neural Network Stochastic Gradient Descent Unsupervised Learning New Initialization Maximum Mean Discrepancy Three Way

January 31, 2023

On the Initialisation of Wide Low-Rank Feedforward Neural Networks
Thiziri Nait Saada, Jared Tanner
Neural Network Low Rank Multiplicative Size Scaling New Initialization Chaotic Dynamic Jacobian Matrix

January 25, 2023

When Layers Play the Lottery, all Tickets Win at Initialization
Artur Jordao, George Correa de Araujo, Helena de Almeida Maia, Helio Pedrini
Deep Network Multi Layer Pruning Method New Initialization Lottery Ticket Sparse Subnetworks Dense Network Robust Ticket Layer Pruning

December 4, 2022

Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channels
Kangyu Weng, Aohua Cheng, Ziyang Zhang, Pei Sun, Yang Tian
Deep Neural Network Information Bottleneck New Initialization Information Propagation Mutual Information Maximization Statistical Mechanic Optimal Channel

October 14, 2022

Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning
John Nguyen, Jianyu Wang, Kshitiz Malik, Maziar Sanjabi, Michael Rabbat
Global Impact Pre Trained Pre Training New Initialization Unobserved Heterogeneity System Heterogeneity

September 15, 2022

Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)
Zhenyu Zhu, Fanghui Liu, Grigorios G Chrysos, Volkan Cevher
Deep Learning Deep Neural Network Native Robustness Large Depth New Initialization Two Layer ReLU Minimum Width Average Case Robustness

September 13, 2022

One-shot Network Pruning at Initialization with Discriminative Image Patches
Yinan Yang, Yu Wang, Ying Ji, Heng Qi, Jien Kato
New Initialization Network Pruning Patch Level Pruning Performance ImageNet Benchmark

September 6, 2022

What to Prune and What Not to Prune at Initialization
Maham Haroon
Edge Pruning Sparse Network New Initialization Sparse Matrix Pruning Framework

August 30, 2022

Persistence Initialization: A novel adaptation of the Transformer architecture for Time Series Forecasting
Espen Haugsdal, Erlend Aune, Massimiliano Ruocco
Transformer Architecture Transformer Model Time Series Forecasting Normalization Dictionary New Initialization Autoregressive Neural Network

July 28, 2022

Initialization and Alignment for Adversarial Texture Optimization
Xiaoming Zhao, Zhizhen Zhao, Alexander G. Schwing
Alignment Problem New Initialization Image Alignment Texture Generation Adversarial Texture

June 30, 2022

Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning
John Nguyen, Jianyu Wang, Kshitiz Malik, Maziar Sanjabi, Michael Rabbat
Global Impact Pre Trained Pre Training New Initialization System Heterogeneity

June 27, 2022

AutoInit: Automatic Initialization via Jacobian Tuning
Tianyu He, Darshil Doshi, Andrey Gromov
Deep Neural Network New Initialization Better Initialization Black Box Tuning Optimal Initialization

New Initialization

Papers

Pruning at Initialization -- A Sketching Perspective

FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models

BO-ICP: Initialization of Iterative Closest Point Based on Bayesian Optimization

Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training

Effective Theory of Transformers at Initialization

On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networks

Does "Deep Learning on a Data Diet" reproduce? Overall yes, but GraNd at Initialization does not

Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initialization

Unsupervised Learning of Initialization in Deep Neural Networks via Maximum Mean Discrepancy

On the Initialisation of Wide Low-Rank Feedforward Neural Networks

When Layers Play the Lottery, all Tickets Win at Initialization

Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channels

Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning

Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)

One-shot Network Pruning at Initialization with Discriminative Image Patches

What to Prune and What Not to Prune at Initialization

Persistence Initialization: A novel adaptation of the Transformer architecture for Time Series Forecasting

Initialization and Alignment for Adversarial Texture Optimization

Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning

AutoInit: Automatic Initialization via Jacobian Tuning