Training Algorithm

Training algorithms are crucial for optimizing machine learning models, aiming to achieve high accuracy and efficiency. Current research emphasizes improving training speed and stability across diverse model architectures, including neural networks (e.g., ResNets, Transformers, and unfolded GNNs), and addressing challenges like local minima, data heterogeneity, and the need for formal guarantees in safety-critical applications. These advancements are vital for scaling machine learning to larger datasets and more complex tasks, impacting fields ranging from image recognition and natural language processing to control systems and scientific modeling.

Papers

February 1, 2023

Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection
Chenglong Wang, Yi Lu, Yongyu Mu, Yimin Hu, Tong Xiao, Jingbo Zhu
Knowledge Distillation Pre Trained Language Model Actor Critic Teacher Model Training Algorithm Knowledge Selection

January 28, 2023

A Closer Look at Few-shot Classification Again
Xu Luo, Hao Wu, Ji Zhang, Lianli Gao, Jing Xu, Jingkuan Song
Transfer Learning Pre Trained Model Shot Classification Glance Annotation Visual Representation Learning Training Algorithm

November 28, 2022

PyTorch Adapt
Kevin Musgrave, Serge Belongie, Ser-Nam Lim
Domain Adaptation Machine Learning Algorithm Pytorch Model Training Algorithm

November 21, 2022

Self-Adaptive, Dynamic, Integrated Statistical and Information Theory Learning
Zsolt János Viharos, Ágnes Szűcs
Neural Network Information Theoretic Self Adaptation Dynamic Learning Training Algorithm Error Metric Levenberg Marquardt

November 2, 2022

RCD-SGD: Resource-Constrained Distributed SGD in Heterogeneous Environment via Submodular Partitioning
Haoze He, Parijat Dube
Stochastic Gradient Descent Submodular Maximization Training Algorithm Heterogeneous Environment SGD Style Inter Class Feature

October 25, 2022

Data-Efficient Model Learning for Control with Jacobian-Regularized Dynamic-Mode Decomposition}
Brian E. Jackson, Jeong Hun Lee, Kevin Tracy, Zachary Manchester
External Control Predictive Control Model Predictive Control Koopman Operator Dynamic Mode Decomposition Training Algorithm

July 7, 2022

Multisymplectic Formulation of Deep Learning Using Mean--Field Type Control and Nonlinear Stability of Training Algorithm
Nader Ganaba
Deep Learning Deep Neural Network Mean Field Training Algorithm Text Control Nonlinear Stability

May 26, 2022

Learning ReLU networks to high uniform accuracy is intractable
Julius Berner, Philipp Grohs, Felix Voigtlaender
Loss Function Generalization Error Statistical Learning Theory Deep ReLU Network Training Algorithm Learning Problem High Precision

March 13, 2022

Training Protocol Matters: Towards Accurate Scene Text Recognition via Training Protocol Searching
Xiaojie Chu, Yongtao Wang, Chunhua Shen, Jingdong Chen, Wei Chu
Scene Text Recognition Training Method Training Algorithm Optimal Training

November 15, 2021

Mitigating Divergence of Latent Factors via Dual Ascent for Low Latency Event Prediction Models
Alex Shtoff, Yair Koren
Real World Latent Factor Training Algorithm Incremental Training Inherent Discontinuity Ad Placement

Training Algorithm

Papers

Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection

A Closer Look at Few-shot Classification Again

PyTorch Adapt

Self-Adaptive, Dynamic, Integrated Statistical and Information Theory Learning

RCD-SGD: Resource-Constrained Distributed SGD in Heterogeneous Environment via Submodular Partitioning

Data-Efficient Model Learning for Control with Jacobian-Regularized Dynamic-Mode Decomposition}

Multisymplectic Formulation of Deep Learning Using Mean--Field Type Control and Nonlinear Stability of Training Algorithm

Learning ReLU networks to high uniform accuracy is intractable

Training Protocol Matters: Towards Accurate Scene Text Recognition via Training Protocol Searching

Mitigating Divergence of Latent Factors via Dual Ascent for Low Latency Event Prediction Models