Optimal Generalization

Optimal generalization in machine learning focuses on developing models that accurately predict outcomes on unseen data, a crucial aspect for reliable AI applications. Current research investigates various optimization techniques, including sharpness-aware minimization and exponential moving averages, and explores the impact of model architecture (depth vs. width), pre-training strategies (like federated meta-learning), and training hyperparameters (e.g., learning rate decay) on generalization performance. These efforts aim to improve the robustness and reliability of machine learning models across diverse domains, from medical image analysis to natural language processing, by mitigating overfitting and enhancing the ability to generalize to new, unseen data.

Papers

October 17, 2024

LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch
Caigao Jiang, Xiang Shu, Hong Qian, Xingyu Lu, Jun Zhou, Aimin Zhou, Yang Yu
Optimization Purpose LLM Based Optimization Problem Scratch Project Definition Defining Redefinition Optimal Generalization

August 7, 2024

Do Sharpness-based Optimizers Improve Generalization in Medical Image Analysis?
Mohamed Hassan, Aleksandar Vakanski, Min Xian
Medical Image Analysis Sharpness Aware Minimization Optimal Generalization Sharpness Measure Sharpness Aware Optimization

June 17, 2024

Pre-Training and Personalized Fine-Tuning via Over-the-Air Federated Meta-Learning: Convergence-Generalization Trade-Offs
Haifeng Wen, Hong Xing, Osvaldo Simeone
Strong Generalization Fine Tuning Pre Training Generalization Performance Air Federated Learning Training Paradigm Optimal Generalization pFL Method

February 14, 2024

Switch EMA: A Free Lunch for Better Flatness and Sharpness
Siyuan Li, Zicheng Liu, Juanxi Tian, Ge Wang, Zedong Wang, Weiyang Jin, Di Wu, Cheng Tan, Tao Lin, Yang Liu, Baigui Sun, Stan Z. Li
Deep Neural Network Free Lunch Deep Neural Network Training Node Feature Sharpness Optimal Generalization Document Flattening Exponential Moving Average

January 31, 2024

Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss
Yahong Yang, Juncai He
Neural Network Physic Informed Neural Network Visual Perspective Deeper Network Wide Neural Network Deep Depth Optimal Generalization Sobolev Training

January 21, 2024

Understanding the Generalization Benefits of Late Learning Rate Decay
Yinuo Ren, Chao Ma, Lexing Ying
Learning Rate Better Generalization Large Learning Rate Optimal Generalization Generalization Benefit

October 4, 2023

Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data
Zhiwei Xu, Yutong Wang, Spencer Frei, Gal Vardi, Wei Hu
Gradient Descent Model Overfitting ReLU Network Benign Overfitting Grokking Phenomenon Optimal Generalization Generalization Behavior

August 26, 2023

Late Stopping: Avoiding Confidently Learning from Mislabeled Examples
Suqin Yuan, Lei Feng, Tongliang Liu
Noisy Label Early Stopping Real World Noisy Datasets Optimal Generalization Unlabeled Example

July 20, 2023

Long-Tail Theory under Gaussian Mixtures
Arman Bolatov, Maxat Tezekbayev, Igor Melnykov, Artur Pak, Vassilina Nikoulina, Zhenisbek Assylbekov
Gaussian Mixture Model Gaussian Mixture Linear Classifier Non Linear Long Tailed Distribution Optimal Generalization

July 11, 2023

Fundamental limits of overparametrized shallow neural networks for supervised learning
Francesco Camilli, Daria Tieplova, Jean Barbier
Neural Network Supervised Learning Neural Network Model Information Theoretic Two Layer Neural Network Shallow Neural Network Fundamental Limit Optimal Generalization

May 30, 2023

Improving Generalization of Complex Models under Unbounded Loss Using PAC-Bayes Bounds
Xitong Zhang, Avrajit Ghosh, Guangliang Liu, Rongrong Wang
Deep Neural Network Gradient Descent Adam Optimizer PAC Bayesian Unlocking Insight Optimal Generalization Inductive Prior

May 27, 2023

Approximation-Generalization Trade-offs under (Approximate) Group Equivariance
Mircea Petrache, Shubhendu Trivedi
Average Approximation Equivariant Neural Network Optimal Generalization Group Equivariance Symmetric Model

February 22, 2023

Learning to Generalize Provably in Learning to Optimize
Junjie Yang, Tianlong Chen, Mingkang Zhu, Fengxiang He, Dacheng Tao, Yingbin Liang, Zhangyang Wang
LeArning Abstract Strong Generalization Learned Optimizers Optimal Generalization

October 5, 2022

Spectral Regularization Allows Data-frugal Learning over Combinatorial Spaces
Amirali Aghazadeh, Nived Rajaraman, Tony Tu, Kannan Ramchandran
Generalization Performance Improved Generalization Learned Function Spectral Regularization Optimal Generalization Frugal Learning High Dimensional Combinatorial

August 17, 2022

Investigating the Impact of Model Width and Density on Generalization in Presence of Label Noise
Yihao Xue, Kyle Whitecross, Baharan Mirzasoleiman
Strong Generalization Noisy Label Label Noise Speech Presence Model Size Double Descent Particle Density Optimal Generalization

June 20, 2022

When Does Re-initialization Work?
Sheheryar Zaidi, Tudor Berariu, Hyunjik Kim, Jörg Bornschein, Claudia Clopath, Yee Whye Teh, Razvan Pascanu
Strong Generalization Optimal Generalization Parameter Re Initialization

June 10, 2022

Intrinsic dimensionality and generalization properties of the $\mathcal{R}$-norm inductive bias
Navid Ardeshir, Daniel Hsu, Clayton Sanford
Inductive Bias Two Layer Neural Network Generalization Property P$ Norm Optimal Generalization Intrinsic Dimensionality

April 6, 2022

Fundamental limits to learning closed-form mathematical models from data
Oscar Fajardo-Fontiveros, Ignasi Reichardt, Harry R. De Los Rios, Jordi Duch, Marta Sales-Pardo, Roger Guimera
Strong Generalization Raw Data Model Learning Fundamental Limit Optimal Generalization Bayesian Model Selection Closed Form

March 24, 2022

Rich Feature Construction for the Optimization-Generalization Dilemma
Jianyu Zhang, David Lopez-Paz, Léon Bottou
Strong Generalization Inductive Bias Distribution Generalization Hyperparameter Tuning Optimal Generalization

November 28, 2021

Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Nets
Shao-Bo Lin, Yao Wang, Ding-Xuan Zhou
Generalization Performance Empirical Risk Minimization Deep ReLU Network Optimal Generalization