Model Initialization

Model initialization, the process of setting initial weights for neural networks, significantly impacts training efficiency and model performance. Current research focuses on improving initialization strategies through techniques like leveraging pretrained models (e.g., transferring weights from larger networks or using ImageNet-pretrained weights), employing quasirandom number generators for weight assignment, and developing meta-learning approaches that learn effective initializations across diverse tasks. These advancements aim to enhance model robustness, generalization, and training speed, ultimately leading to more efficient and effective machine learning applications across various domains, including medical image analysis and federated learning.

Papers

October 18, 2024

2D-3D Deformable Image Registration of Histology Slide and Micro-CT with ML-based Initialization
Junan Chen, Matteo Ronchetti, Verena Stehl, Van Nguyen, Muhannad Al Kallaa, Mahesh Thalwaththe Gedara, Claudia Lölkes, Stefan Moser, Maximilian Seidl, Matthias Wieczorek
Deformable Image Registration Multi Modal Image Histology Slide Model Initialization

September 13, 2024

Rethinking Meta-Learning from a Learning Lens
Jingyao Wang, Wenwen Qiang, Jiangmeng Li, Lingyu Si, Changwen Zheng
Meta Learning Task Specific Task Relatedness Task Learning Model Initialization

August 5, 2024

On Using Quasirandom Sequences in Machine Learning for Model Weight Initialization
Andriy Miranskyy, Adam Sorrenti, Viral Thakar
Machine Learning Random Sequence Random Number Generator Weight Initialization Model Initialization Driven Initialization

March 11, 2024

XB-MAML: Learning Expandable Basis Parameters for Effective Meta-Learning with Wide Task Coverage
Jae-Jun Lee, Sung Whan Yoon
Meta Learning Initialization Bias Model Initialization Basis Expansion Meta Learning Benchmark

November 30, 2023

Initializing Models with Larger Ones
Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov, Yida Yin, Zhiqiang Shen, Trevor Darrell, Lingjie Liu, Zhuang Liu
Neural Network Knowledge Distillation Weight Initialization Model Initialization Weight Selection

October 31, 2023

A Comprehensive Study on Model Initialization Techniques Ensuring Efficient Federated Learning
Ishmeet Kaur, Adwaita Janardhan Jadhav
Federated Learning Federated Prompt Cooperation Comprehensive Study Model Initialization

September 20, 2023

Uncovering the effects of model initialization on deep model generalization: A study with adult and pediatric Chest X-ray images
Sivaramakrishnan Rajaraman, Ghada Zamzmi, Feng Yang, Zhaohui Liang, Zhiyun Xue, Sameer Antani
Mixed Effect Neural Network Generalization Older Adult Weighted Ensemble Model Initialization Pre Trained Model Weight Pediatric CXR

July 31, 2023

Boosting Adverse Drug Event Normalization on Social Media: General-Purpose Model Initialization and Biomedical Semantic Text Similarity Benefit Zero-Shot Linking in Informal Contexts
François Remy, Simone Scaboro, Beatrice Portelli
Zero Shot Social Medium Semantic Textual Similarity Biomedical Entity Linking Entity Normalization Adverse Drug Informal Language Model Initialization

July 25, 2023

Scaling up machine learning-based chemical plant simulation: A method for fine-tuning a model to induce stable fixed points
Malte Esders, Gimmy Alex Fernandez Ramirez, Michael Gastegger, Satya Swarup Samal
Machine Learning Model Full Model New Machine Practical Method Exceptional Point First Principle Multiple Model Model Initialization

May 26, 2023

A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models
Hayeon Lee, Rui Hou, Jongpil Kim, Davis Liang, Sung Ju Hwang, Alexander Min
Knowledge Distillation Pre Trained Language Model Study Feature Teacher Model Multi Teacher Distillation Model Initialization

October 13, 2022

Attribution-aware Weight Transfer: A Warm-Start Initialization for Class-Incremental Semantic Segmentation
Dipam Goswami, René Schuster, Joost van de Weijer, Didier Stricker
Foreground Object Warm Start Incremental Semantic Segmentation Class Incremental Semantic Model Initialization Weight Transfer Background Shift

August 5, 2022

Parameter Averaging for Feature Ranking
Talip Ucar, Ehsan Hajiramezanali
Ground Truth Feature Importance Binary Mask Feature Ranking Model Initialization Parameter Averaging

Model Initialization

Papers

2D-3D Deformable Image Registration of Histology Slide and Micro-CT with ML-based Initialization

Rethinking Meta-Learning from a Learning Lens

On Using Quasirandom Sequences in Machine Learning for Model Weight Initialization

XB-MAML: Learning Expandable Basis Parameters for Effective Meta-Learning with Wide Task Coverage

Initializing Models with Larger Ones

A Comprehensive Study on Model Initialization Techniques Ensuring Efficient Federated Learning

Uncovering the effects of model initialization on deep model generalization: A study with adult and pediatric Chest X-ray images

Boosting Adverse Drug Event Normalization on Social Media: General-Purpose Model Initialization and Biomedical Semantic Text Similarity Benefit Zero-Shot Linking in Informal Contexts

Scaling up machine learning-based chemical plant simulation: A method for fine-tuning a model to induce stable fixed points

A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models

Attribution-aware Weight Transfer: A Warm-Start Initialization for Class-Incremental Semantic Segmentation

Parameter Averaging for Feature Ranking