Generalization Error

Generalization error, the difference between a model's performance on training and unseen data, is a central challenge in machine learning. Current research focuses on understanding and mitigating this error across various model architectures, including linear models, neural networks (especially deep and overparameterized ones), and graph neural networks, often employing techniques like stochastic gradient descent, early stopping, and ensemble methods such as bagging. This research aims to develop tighter theoretical bounds on generalization error and improve model selection and assessment, particularly under conditions like data scarcity, distribution shifts, and adversarial attacks. Improved understanding of generalization error is crucial for building more reliable and robust machine learning systems across diverse applications.

Papers

January 5, 2024

Class-wise Generalization Error: an Information-Theoretic Analysis
Firas Laakom, Yuheng Bu, Moncef Gabbouj
Strong Generalization Generalization Performance Generalization Bound Generalization Error

January 3, 2024

Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decay
Yicheng Li, Weiye Gan, Zuoqiang Shi, Qian Lin
Neural Tangent Kernel Generalization Error Kernel Regression Kernel Based Spectral Algorithm

December 25, 2023

November 29, 2023

Critical Influence of Overparameterization on Sharpness-aware Minimization
Sungbin Shin, Dongyeop Lee, Maksym Andriushchenko, Namhoon Lee
Generalization Error Sharpness Aware Minimization Overparametrization Bound Mutual Influence

November 28, 2023

On robust overfitting: adversarial training induced distribution matters
Runzhi Tian, Yongyi Mao
Adversarial Training Loss Function Generalization Error Product Distribution Robust Overfitting

October 29, 2023

A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning
Alicia Curth, Alan Jeffares, Mihaela van der Schaar
Many Parameter Generalization Error Statistical Learning Double Descent Model Complexity Multivariate Complexity

October 26, 2023

Optimization dependent generalization bound for ReLU networks based on sensitivity in the tangent bundle
Dániel Rácz, Mihály Petreczky, András Csertán, Bálint Daróczy
Deep Learning Deep Neural Network Optimization Purpose Gradient Descent Generalization Error ReLU Network Network Sensitivity Tangent Bundle

October 23, 2023

Evaluating machine learning models in non-standard settings: An overview and new findings
Roman Hornung, Malte Nalenz, Lennart Schneider, Andreas Bender, Ludwig Bothmann, Bernd Bischl, Thomas Augustin, Anne-Laure Boulesteix
Machine Learning Generalization Error New Finding Resampling Method

October 13, 2023

It's an Alignment, Not a Trade-off: Revisiting Bias and Variance in Deep Models
Lin Chen, Michal Lukasik, Wittawat Jitkrittum, Chong You, Sanjiv Kumar
Deep Learning Model Alignment Problem Deep Model Generalization Error Neural Collapse Variance Information

September 20, 2023

PAGER: A Framework for Failure Analysis of Deep Regression Models
Jayaraman J. Thiagarajan, Vivek Narayanaswamy, Puja Trivedi, Rushil Anirudh
New Framework Deep Model Generalization Error Deep Regression Failure Analysis

September 10, 2023

Generalization error bounds for iterative learning algorithms with bounded updates
Jingwen Fu, Nanning Zheng
Generalization Error Information Theoretic Iterative Training Non Convex Loss Function Information System Update Generalization Theory

August 16, 2023

Two Phases of Scaling Laws for Nearest Neighbor Classifiers
Pengkun Yang, Jingzhao Zhang
Generalization Error Scaling Law Civil Engineering Phase Nearest Neighbor Classifier

July 25, 2023

Modify Training Directions in Function Space to Reduce Generalization Error
Yi Yu, Wenlian Lu, Boyu Chen
Neural Tangent Kernel Generalization Error Function Space Natural Gradient Descent

July 20, 2023

Flatness-Aware Minimization for Domain Generalization
Xingxuan Zhang, Renzhe Xu, Han Yu, Yancheng Dong, Pengfei Tian, Peng Cu
Domain Generalization Generalization Error Flatness Aware

July 10, 2023

Generalization Error of First-Order Methods for Statistical Learning with Generic Oracles
Kevin Scaman, Mathieu Even, Laurent Massoulié
Generalization Error Statistical Learning Non Smooth Mini Batch Gradient Descent Strongly Convex First Order Method

July 5, 2023

GIT: Detecting Uncertainty, Out-Of-Distribution and Adversarial Samples using Gradients and Invariance Transformations
Julia Lust, Alexandru P. Condurache
Deep Neural Network Adversarial Attack Strong Generalization High Uncertainty Anticipation Distribution Data Adversarial Sample Generalization Error Transformation Invariance

June 23, 2023

A new approach to generalisation error of machine learning algorithms: Estimates and convergence
Michail Loulakis, Charalambos G. Makridakis
Neural Network Deep Learning Estimation Task Early Stage Convergence Novel Approach Machine Learning Algorithm Generalization Error Learned Function Neural Interpolation

June 20, 2023

Mean-field Analysis of Generalization Errors
Gholamali Aminian, Samuel N. Cohen, Łukasz Szpruch
Supervised Learning Empirical Risk Minimization Mean Field Generalization Error Regularity Condition

June 9, 2023

Lessons from Generalization Error Analysis of Federated Learning: You May Communicate Less Often!
Milad Sefidgaran, Romain Chor, Abdellatif Zaidi, Yijun Wan
Strong Generalization Loss Function Generalization Error Critical Lesson Generalization Capacity