Generalization Error

Generalization error, the difference between a model's performance on training and unseen data, is a central challenge in machine learning. Current research focuses on understanding and mitigating this error across various model architectures, including linear models, neural networks (especially deep and overparameterized ones), and graph neural networks, often employing techniques like stochastic gradient descent, early stopping, and ensemble methods such as bagging. This research aims to develop tighter theoretical bounds on generalization error and improve model selection and assessment, particularly under conditions like data scarcity, distribution shifts, and adversarial attacks. Improved understanding of generalization error is crucial for building more reliable and robust machine learning systems across diverse applications.

Papers

June 7, 2024

Error Bounds of Supervised Classification from Information-Theoretic Perspective
Binchuan Qi
Deep Learning Deep Neural Network Generalization Error Information Theoretic Error Bound Generalization Capability Supervised Classification

May 30, 2024

FlexiDrop: Theoretical Insights and Practical Advances in Random Dropout Method on GNNs
Zhiheng Zhou, Sihao Liu, Weichen Zhao
Graph Neural Network Generalization Error Theoretical Analysis Practical Application

May 24, 2024

Information-theoretic Generalization Analysis for Expected Calibration Error
Futoshi Futami, Masahiro Fujisawa
Calibration Performance Generalization Error Bias Evaluation Calibration Error Estimation Bias Information Theoretic Generalization

May 23, 2024

Bagging Improves Generalization Exponentially
Huajie Qian, Donghao Ying, Henry Lam, Wotao Yin
Generalization Performance Empirical Risk Minimization Robust Optimization Generalization Error Stable Bagging Fast Generalization

May 15, 2024

Restoring balance: principled under/oversampling of data for optimal classification
Emanuele Loffredo, Mauro Pastore, Simona Cocco, Rémi Monasson
Raw Data Class Imbalance Generalization Error Balancing Strategy Linear Classifier Oversampling Method Optimal Classifier

April 14, 2024

Generalization Error Bounds for Learning under Censored Feedback
Yifan Yang, Ali Payani, Parinaz Naghizadeh
Generalization Error Error Bound Generalization Error Bound Generalization Guarantee

April 7, 2024

The Sample Complexity of Gradient Descent in Stochastic Convex Optimization
Roi Livni
Gradient Descent Sample Complexity Empirical Risk Minimization Generalization Error Stochastic Convex Optimization Full Batch Gradient Descent Smooth Stochastic Convex Optimization

March 26, 2024

Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
Jinze Zhao, Peihao Wang, Zhangyang Wang
Many Sparse Sparsity Increase Generalization Error Preliminary Study Sparse Mixture of Expert Expert Selection Classical Learning

March 18, 2024

Generalization error of spectral algorithms
Maksim Velikanov, Maxim Panov, Dmitry Yarotsky
Generalization Error Kernel Method Kernel Ridge Regression Gaussian Distribution Spectral Algorithm Unbounded Loss

March 14, 2024

Upper Bound of Bayesian Generalization Error in Partial Concept Bottleneck Model (CBM): Partial CBM outperforms naive CBM
Naoki Hayashi, Yoshihide Sawada
Generalization Performance Generalization Error Concept Bottleneck Model

February 27, 2024

Variational Learning is Effective for Large Deep Networks
Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, Thomas Möllenhoff
Neural Network Generalization Error Variational Learning Online Newton

February 24, 2024

A priori Estimates for Deep Residual Network in Continuous-time Reinforcement Learning
Shuyu Yin, Qixuan Zhou, Fei Wen, Tao Luo
Reinforcement Learning Loss Function Generalization Error Deep Reinforcement Residual Network Control Problem Prior Value Estimate

February 19, 2024

The effect of Leaky ReLUs on the training and generalization of overparameterized networks
Yinglong Guo, Shaohan Li, Gilad Lerman
Neural Network Strong Generalization Training Data Mixed Effect Network Programming Generalization Error Leaky ReLU

February 15, 2024

Random features and polynomial rules
Fabián Aguirre-López, Silvio Franz, Mauro Pastore
Machine Learning Generalization Error Random Feature Polynomial System

February 13, 2024

Model Assessment and Selection under Temporal Distribution Shift
Elise Han, Chengpiao Huang, Kaizheng Wang
Generalization Error Model Selection View Selection Model Evaluation Temporal Distribution Shift Window Approach

February 10, 2024

Generalization Error of Graph Neural Networks in the Mean-field Regime
Gholamali Aminian, Yixuan He, Gesine Reinert, Łukasz Szpruch, Samuel N. Cohen
Neural Network Convolutional Neural Network Graph Neural Network Mean Field Generalization Error Parameterized Regime Graph Sample

February 4, 2024

Early stopping by correlating online indicators in neural networks
Manuel Vilares Ferro, Yerai Doval Mosquera, Francisco J. Ribadas Pena, Victor M. Darriba Bilbao
Neural Network Machine Learning Generalization Error Model Overfitting Cross Validation Early Stopping Aggregate Level Indicator

February 2, 2024

Parameter uncertainties for imperfect surrogate models in the low-noise regime
Thomas D Swinburne, Danny Perez
Surrogate Model Generalization Error Posterior Distribution Parameter Uncertainty Surrogate Modelling Bayesian Regression

January 23, 2024

The twin peaks of learning neural networks
Elizaveta Demyanenko, Christoph Feinauer, Enrico M. Malatesta, Luca Saglietti
Neural Network Adversarial Attack Generalization Error Model Overfitting Dimensionless Variable

January 19, 2024

Generalization Error Guaranteed Auto-Encoder-Based Nonlinear Model Reduction for Operator Learning
Hao Liu, Biraj Dahal, Rongjie Lai, Wenjing Liao
Latent Variable Generalization Error Operator Learning Model Reduction Low Dimensional Structure Nonlinear Model Reduction