Generalization Error

Generalization error, the difference between a model's performance on training and unseen data, is a central challenge in machine learning. Current research focuses on understanding and mitigating this error across various model architectures, including linear models, neural networks (especially deep and overparameterized ones), and graph neural networks, often employing techniques like stochastic gradient descent, early stopping, and ensemble methods such as bagging. This research aims to develop tighter theoretical bounds on generalization error and improve model selection and assessment, particularly under conditions like data scarcity, distribution shifts, and adversarial attacks. Improved understanding of generalization error is crucial for building more reliable and robust machine learning systems across diverse applications.

Papers

March 21, 2023

Blow-up Algorithm for Sum-of-Products Polynomials and Real Log Canonical Thresholds
Joe Hirose
Generalization Error Polynomial System Canonical Threshold Blow Up Algorithm

March 17, 2023

Deep Nonparametric Estimation of Intrinsic Data Structures by Chart Autoencoders: Generalization Error and Robustness
Hao Liu, Alex Havrilla, Rongjie Lai, Wenjing Liao
Native Robustness Supervised Autoencoder Generalization Error Low Dimensional Manifold Autoencoder Based Neural Network Autoencoder Data Structure Classical Autoencoders Deep Nonparametric

March 16, 2023

Bayesian Generalization Error in Linear Neural Networks with Concept Bottleneck Structure and Multitask Formulation
Naoki Hayashi, Yoshihide Sawada
Multi Task Generalization Performance Generalization Error Concept Bottleneck Model Linear Neural Network Three Layer Neural Network

March 10, 2023

March 2, 2023

Over-training with Mixup May Hurt Generalization
Zixuan Liu, Ziqiao Wang, Hongyu Guo, Yongyi Mao
Strong Generalization Generalization Error Local Mixup Mixup Training LabEl Noise Speech Text Manifold Mixup

February 25, 2023

Generalization Bounds for Set-to-Set Matching with Negative Sampling
Masanari Kimura
Generalization Bound Generalization Error Negative Sampling Black Box Function Set to Set

February 18, 2023

The Generalization Error of Stochastic Mirror Descent on Over-Parametrized Linear Models
Danil Akhtiamov, Babak Hassibi
Stochastic Gradient Descent Generalization Error Parameterized Model Stochastic Mirror Descent

February 16, 2023

Learning-based solutions to nonlinear hyperbolic PDEs: Empirical insights on generalization errors
Bilal Thonnam Thodi, Sai Venkata Ramana Ambadipudi, Saif Eddin Jabari
Empirical Study Solution Path Generalization Error Fourier Neural Operator Dual Formulation Hyperbolic Partial Differential Equation Physic Based Regularization

February 12, 2023

Theory on Forgetting and Generalization of Continual Learning
Sen Lin, Peizhong Ju, Yingbin Liang, Ness Shroff
Strong Generalization Continual LEArning Catastrophic Forgetting Theoretical Understanding Generalization Performance Anti Forgetting Generalization Error

January 28, 2023

On the Lipschitz Constant of Deep Networks and Double Descent
Matteo Gamba, Hossein Azizpour, Mårten Björkman
Deep Network Generalization Error Implicit Regularization Double Descent Model Complexity Lipschitz Bound Input Gradient

December 22, 2022

A Mathematical Framework for Learning Probability Distributions
Hongkang Yang
Strong Generalization Generative Modeling Generalization Error Density Estimation Distribution Learning Learning Probability Distribution

December 13, 2022

December 2, 2022

On the Change of Decision Boundaries and Loss in Learning with Concept Drift
Fabian Hinder, Valerie Vaquet, Johannes Brinkrolf, Barbara Hammer
LeArning Abstract Generalization Error Concept Drift Decision Boundary Pre Change Information Drift Detection Interest Loss Real Drift Interleaved Learning

November 15, 2022

Quantifying the Impact of Label Noise on Federated Learning
Shuqi Ke, Chao Huang, Xin Liu
Global Impact Label Noise Generalization Error Global Model

November 12, 2022

Empirical Risk Minimization with Relative Entropy Regularization
Samir M. Perlaza, Gaetan Bisson, Iñaki Esnaola, Alain Jean-Marie, Stefano Rini
Empirical Risk Minimization Generalization Error Entropy Regularization Sub Gaussian Gibbs Algorithm

October 31, 2022

Class Interference of Deep Neural Networks
Dongcui Diao, Hengshuai Yao, Bei Jiang
Deep Neural Network Deep Network Generalization Error Network Interference Class Interference

October 24, 2022

A PAC-Bayesian Generalization Bound for Equivariant Networks
Arash Behboodi, Gabriele Cesa, Taco Cohen
Generalization Error Inter Part Equivariance Equivariant Network PAC Bayesian Generalization Bound Group Representation

October 15, 2022

How Does Pseudo-Labeling Affect the Generalization Error of the Semi-Supervised Gibbs Algorithm?
Haiyun He, Gholamali Aminian, Yuheng Bu, Miguel Rodrigues, Vincent Y. F. Tan
Semi Supervised Learning Pseudo Labeling Generalization Error Gibbs Algorithm Pseudo Labeled