New Initialization

New initialization techniques for neural networks aim to improve training efficiency, stability, and generalization performance by carefully selecting initial model parameters. Current research focuses on developing methods tailored to specific architectures like transformers and diffusion models, often leveraging techniques such as reparameterization, knowledge factorization, and adaptive segmentation to optimize initialization for various tasks, including image generation, natural language processing, and visual navigation. These advancements are significant because they can lead to faster training, reduced computational costs, and improved model accuracy across a wide range of applications.

Papers

February 2, 2024

No Free Prune: Information-Theoretic Barriers to Pruning at Initialization
Tanishq Kumar, Kevin Luo, Mark Sellke
Mutual Information Edge Pruning Information Theoretic Sparse Network New Initialization Sparse Neural Network Sparse Subnetworks Dense Model

January 23, 2024

Maximizing Data Efficiency for Cross-Lingual TTS Adaptation by Self-Supervised Representation Mixing and Embedding Initialization
Wei-Ping Huang, Sung-Feng Huang, Hung-yi Lee
Text to Speech New Initialization Language Adaptation Data Efficiency Effective Transfer Learning Domain Mixing Text to Speech System

December 10, 2023

Initialization Matters for Adversarial Transfer Learning
Andong Hua, Jindong Gu, Zhiyu Xue, Nicholas Carlini, Eric Wong, Yao Qin
Adversarial Robustness New Initialization Finetune Paradigm Adversarial Fine Tuning Robust Pre Adversarial Transfer

December 5, 2023

On the Initialization of Graph Neural Networks
Jiahang Li, Yakun Song, Xiang Song, David Paul Wipf
Graph Neural Network New Initialization GNN Layer

November 24, 2023

ARIA: On the Interaction Between Architectures, Initialization and Aggregation Methods for Federated Visual Classification
Vasilis Siomos, Sergio Naval-Marimont, Jonathan Passerat-Palmbach, Giacomo Tarroni
Supervised ImageNet New Initialization Architecture Perspective Normalization Layer Federated Image Classification Medical Image Classification Task Diva 360

November 6, 2023

Initialisation of Autonomous Aircraft Visual Inspection Systems via CNN-Based Camera Pose Estimation
Xueyan Oh, Leonard Loh, Shaohui Foong, Zhong Bao Andy Koh, Kow Leong Ng, Poh Kang Tan, Pei Lin Pearlin Toh, U-Xuan Tan
Deep Convolutional Neural Network Camera Pose New Initialization Visual Inspection Localization Method Visual Quality Inspection

October 31, 2023

Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networks
Jiayuan Ye, Zhenyu Zhu, Fanghui Liu, Reza Shokri, Volkan Cevher
Neural Network New Initialization Data Leakage Benign Over Parameterization Privacy Utility Trade

October 3, 2023

How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initialization
Nuoya Xiong, Lijun Ding, Simon S. Du
Gradient Descent Ground Truth Cyclic Symmetry New Initialization Benign Over Parameterization Matrix Sensing

June 20, 2023

Principles for Initialization and Architecture Selection in Graph Neural Networks with ReLU Activations
Gage DeZoort, Boris Hanin
Graph Neural Network General Principle New Initialization ReLU Activation Residual Connection

May 31, 2023

Sentence Simplification Using Paraphrase Corpus for Initialization
Kang Liu, Jipeng Qiang
Parallel Corpus New Initialization Sentence Simplification Paraphrase Corpus

May 28, 2023

On the impact of activation and normalization in obtaining isometric embeddings at initialization
Amir Joudaki, Hadi Daneshmand, Francis Bach
Deep Neural Network Global Impact Normalization Dictionary New Initialization Targeted Activation Penalty Layer Normalization Normalization Layer Gram Matrix Isometric Representation

May 27, 2023

Pruning at Initialization -- A Sketching Perspective
Noga Bar, Raja Giryes
Practical Algorithm Edge Pruning Sparse Network New Initialization Lottery Ticket Hypothesis Sparse Mask Data Independent Perspective Sketch

May 23, 2023

FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models
Konstantin Dobler, Gerard de Melo
Language Model Multilingual Model High Resource Language Human Driving Focus New Initialization Monolingual Text Sparse Softmax

April 25, 2023

BO-ICP: Initialization of Iterative Closest Point Based on Bayesian Optimization
Harel Biggie, Andrew Beathard, Christoffer Heckman
Point Cloud Bayesian Optimization Point Cloud Registration New Initialization Iterative Closest Point

April 6, 2023

Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training
Luís Carvalho, João Lopes Costa, José Mourão, Gonçalo Oliveira
Training Data Gaussian Process New Initialization Wide Neural Network Random Field Width Network Layer NTK

April 4, 2023

Effective Theory of Transformers at Initialization
Emily Dinan, Sho Yaida, Susan Zhang
Transformer Megatron Decepticons New Initialization Deep Transformer Residual Neural Network Backward Propagation Effective Theory

March 31, 2023

On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networks
Sebastian Neumayer, Lénaïc Chizat, Michael Unser
Gradient Descent Mixed Effect Convex Set Two Layer Neural Network Root to Leaf Path New Initialization Two Layer ReLU Unbalanced Optimal Transport Regularization Path

March 26, 2023

Does "Deep Learning on a Data Diet" reproduce? Overall yes, but GraNd at Initialization does not
Andreas Kirsch
Deep Learning Gradient Norm New Initialization Data Pruning Random Pruning Data Diet

February 20, 2023

Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initialization
Cameron Jakub, Mihai Nica
Neural Network Deep Neural Network ReLU Network New Initialization Tilting Angle Combinatorial Approach Model Degeneracy ReLU Function

February 8, 2023

Unsupervised Learning of Initialization in Deep Neural Networks via Maximum Mean Discrepancy
Cheolhyoung Lee, Kyunghyun Cho
Deep Learning Deep Neural Network Stochastic Gradient Descent Unsupervised Learning New Initialization Maximum Mean Discrepancy Three Way

New Initialization

Papers

No Free Prune: Information-Theoretic Barriers to Pruning at Initialization

Maximizing Data Efficiency for Cross-Lingual TTS Adaptation by Self-Supervised Representation Mixing and Embedding Initialization

Initialization Matters for Adversarial Transfer Learning

On the Initialization of Graph Neural Networks

ARIA: On the Interaction Between Architectures, Initialization and Aggregation Methods for Federated Visual Classification

Initialisation of Autonomous Aircraft Visual Inspection Systems via CNN-Based Camera Pose Estimation

Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networks

How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initialization

Principles for Initialization and Architecture Selection in Graph Neural Networks with ReLU Activations

Sentence Simplification Using Paraphrase Corpus for Initialization

On the impact of activation and normalization in obtaining isometric embeddings at initialization

Pruning at Initialization -- A Sketching Perspective

FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models

BO-ICP: Initialization of Iterative Closest Point Based on Bayesian Optimization

Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training

Effective Theory of Transformers at Initialization

On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networks

Does "Deep Learning on a Data Diet" reproduce? Overall yes, but GraNd at Initialization does not

Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initialization

Unsupervised Learning of Initialization in Deep Neural Networks via Maximum Mean Discrepancy