Robust Initialization

Robust initialization methods aim to improve the training efficiency and stability of various neural network architectures by carefully selecting initial parameter values. Current research focuses on developing initialization strategies tailored to specific models, including ResNets, recurrent neural networks (RNNs), and those used in image processing (e.g., kernel regression methods), often leveraging techniques like segmentation, pre-training, and meta-learning to achieve robustness. These advancements lead to faster convergence, improved generalization, and reduced sensitivity to hyperparameter choices, impacting diverse applications from image processing and autonomous driving to federated learning and scientific computing.

Papers

October 25, 2024

A Robust and Efficient Visual-Inertial Initialization with Probabilistic Normal Epipolar Constraint
Changshi Mu, Daquan Feng, Qi Zheng, Yuan Zhuang
High Efficiency Visual Inertial Odometry Rotation Estimation Epipolar Constraint Robust Initialization Gyroscope Bias Visual Inertial Initialization

September 16, 2024

Adaptive Segmentation-Based Initialization for Steered Mixture of Experts Image Regression
Yi-Hsin Li, Sebastian Knorr, Mårten Sjöström, Thomas Sikora
Adaptive Importance New Initialization Image Regression Robust Initialization Stick Breaking Process Mixture

March 14, 2024

RoDUS: Robust Decomposition of Static and Dynamic Elements in Urban Scenes
Thang-Anh-Quan Nguyen, Luis Roldão, Nathan Piasco, Moussab Bennehar, Dzmitry Tsishkou
Large Scale Scene Dynamic Object Urban Scene Semantic Actor Robust Initialization Robust Decomposition Background Reconstruction

March 12, 2024

Stereo-NEC: Enhancing Stereo Visual-Inertial SLAM Initialization with Normal Epipolar Constraints
Weihan Wang, Chieh Chou, Ganesh Sevagamoorthy, Kevin Chen, Zheng Chen, Ziyue Feng, Youjie Xia, Feiyang Cai, Yi Xu, Philippos Mordohai
Visual SLAM Mono to Stereo Epipolar Constraint Robust Initialization Gyroscope Bias Real World Stereo Signal Visual Inertial Initialization

February 3, 2024

Rethinking the Starting Point: Collaborative Pre-Training for Federated Downstream Tasks
Yun-Wei Chu, Dong-Jun Han, Seyyedali Hosseinalipour, Christopher G. Brinton
Pre Trained Model Downstream Task Pre Training Paradigm Robust Initialization

November 27, 2023

Stability-Informed Initialization of Neural Ordinary Differential Equations
Theodor Westny, Arman Mohammadi, Daniel Jung, Erik Frisk
Neural ODE Neural Ordinary Differential Equation Learned Model Robust Initialization Initialization Scheme

November 7, 2023

Improved weight initialization for deep and narrow feedforward neural network
Hyunwoo Lee, Yunho Kim, Seung Yeop Yang, Hayoung Choi
Neural Network Deep Neural Network ReLU Activation Weight Matrix Weight Initialization Robust Initialization

August 23, 2023

Stabilizing RNN Gradients through Pre-training
Luca Herranz-Celotti, Jean Rouat
Recurrent Network Fully Connected Robust Initialization Neural Network Gradient

June 7, 2023

Training-Free Neural Active Learning with Initialization-Robustness Guarantees
Apivich Hemachandra, Zhongxiang Dai, Jasraj Singh, See-Kiong Ng, Bryan Kian Hsiang Low
Gaussian Process Robust Initialization Neural Active Learning EV GP Criterion

July 27, 2022

PointFix: Learning to Fix Domain Bias for Robust Online Stereo Adaptation
Kwonyoung Kim, Jungin Park, Jiyoung Lee, Dongbo Min, Kwanghoon Sohn
LeArning Abstract Stereo Matching Robust Initialization Domain Bias Selective Neural Network Base Model

June 26, 2022

Bounding the Width of Neural Networks via Coupled Initialization -- A Worst Case Analysis
Alexander Munteanu, Simon Omlor, Zhao Song, David P. Woodruff
Neural Network Neural Tangent Kernel Worst Case General Bound Two Layer ReLU Minimum Width Robust Initialization

June 16, 2022

Large-width asymptotics for ReLU neural networks with $\alpha$-Stable initializations
Stefano Favaro, Sandra Fortini, Stefano Peluchetti
Neural Network Training Dynamic ReLU Layer Stable Neural Network Robust Initialization

May 23, 2022

PCA-Boosted Autoencoders for Nonlinear Dimensionality Reduction in Low Data Regimes
Muhammad Al-Digeil, Yuri Grinberg, Daniele Melati3, Mohsen Kamandar Dezfouli, Jens H. Schmid, Pavel Cheben, Siegfried Janz, Dan-Xia Xu
Supervised Autoencoder Dimensionality Reduction Principal Component Analysis Low Data Regime Robust Initialization

February 2, 2022

Nonlinear Initialization Methods for Low-Rank Neural Networks
Kiran Vodrahalli, Rakesh Shivanna, Maheswaran Sathiamoorthy, Sagar Jain, Ed H. Chi
Neural Network Low Rank Low Rank Matrix Robust Initialization Rank Initialization

December 23, 2021

A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization
Enrico Civitelli, Alessio Sortino, Matteo Lapucci, Francesco Bagattini, Giulio Galvan
Batch Normalization Residual Block ResNet Model Weight Initialization Robust Initialization Normalizer Free

Robust Initialization

Papers

A Robust and Efficient Visual-Inertial Initialization with Probabilistic Normal Epipolar Constraint

Adaptive Segmentation-Based Initialization for Steered Mixture of Experts Image Regression

RoDUS: Robust Decomposition of Static and Dynamic Elements in Urban Scenes

Stereo-NEC: Enhancing Stereo Visual-Inertial SLAM Initialization with Normal Epipolar Constraints

Rethinking the Starting Point: Collaborative Pre-Training for Federated Downstream Tasks

Stability-Informed Initialization of Neural Ordinary Differential Equations

Improved weight initialization for deep and narrow feedforward neural network

Stabilizing RNN Gradients through Pre-training

Training-Free Neural Active Learning with Initialization-Robustness Guarantees

PointFix: Learning to Fix Domain Bias for Robust Online Stereo Adaptation

Bounding the Width of Neural Networks via Coupled Initialization -- A Worst Case Analysis

Large-width asymptotics for ReLU neural networks with $\alpha$-Stable initializations

PCA-Boosted Autoencoders for Nonlinear Dimensionality Reduction in Low Data Regimes

Nonlinear Initialization Methods for Low-Rank Neural Networks

A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization