Generalization Gap

The generalization gap, the difference between a model's training and test performance, is a central challenge in machine learning, hindering the reliable deployment of models in real-world scenarios. Current research focuses on understanding and mitigating this gap across various model architectures, including deep neural networks, graph neural networks, and Boolean networks, employing techniques like sharpness-aware minimization, data augmentation, and information-theoretic approaches. Addressing the generalization gap is crucial for improving the reliability and robustness of machine learning models across diverse applications, from computer vision and natural language processing to reinforcement learning and medical image analysis. A deeper understanding of this gap is essential for building more trustworthy and effective AI systems.

Papers

March 13, 2023

Bayes Complexity of Learners vs Overfitting
Grzegorz Głuch, Rudiger Urbanke
Neural Network LeArning Abstract Complexity Matter Generalization Gap Statistical Complexity Periodic Function

March 4, 2023

The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges
Maria Lymperaiou, Giorgos Stamou
Knowledge Graph Technical Challenge Knowledge Based New Task Client Contribution Generalization Gap Linguistic Knowledge Visual Linguistic

February 9, 2023

Generalization in Graph Neural Networks: Improved PAC-Bayesian Bounds on Graph Diffusion
Haotian Ju, Dongyue Li, Aneesh Sharma, Hongyang R. Zhang
Graph Neural Network Strong Generalization Generalization Bound Generalization Gap Graph Diffusion Graph Isomorphism Network Graph Level Classification PAC Bayesian Bound Message Passing Network

October 21, 2022

A New Perspective for Understanding Generalization Gap of Deep Neural Networks Trained with Large Batch Sizes
Oyebade K. Oyedotun, Konstantinos Papadopoulos, Djamila Aouada
Deep Neural Network New Perspective Generalization Gap Neural Network Generalization Mini Batch Gradient Descent

October 5, 2022

The Calibration Generalization Gap
A. Michael Carrell, Neil Mallinar, James Lucas, Preetum Nakkiran
Strong Generalization Generalization Gap

July 13, 2022

Efficient Augmentation for Imbalanced Deep Learning
Damien Dablain, Colin Bellinger, Bartosz Krawczyk, Nitesh Chawla
Convolutional Neural Network Imbalanced Data Generalization Gap Balanced Dataset Efficient Augmentation Deep Imbalanced

July 12, 2022

RelaxLoss: Defending Membership Inference Attacks without Losing Utility
Dingfan Chen, Ning Yu, Mario Fritz
Membership Inference Attack Task Utility Generalization Gap Confidence Relaxation

July 1, 2022

On Leave-One-Out Conditional Mutual Information For Generalization
Mohamad Rida Rammal, Alessandro Achille, Aditya Golatkar, Suhas Diggavi, Stefano Soatto
Strong Generalization Generalization Bound Generalization Gap Conditional Mutual Information

June 28, 2022

Studying Generalization Through Data Averaging
Carlos A. Gomez-Uribe
Strong Generalization Machine Learning Model Generalization Gap

June 11, 2022

A General framework for PAC-Bayes Bounds for Meta-Learning
Arezou Rezazadeh
General Framework Generalization Gap PAC Bayes Bound

June 9, 2022

Trajectory-dependent Generalization Bounds for Deep Neural Networks via Fractional Brownian Motion
Chengli Tan, Jiangshe Zhang, Junmin Liu
Deep Neural Network Generalization Gap Gradient Noise Rademacher Complexity Fractional Brownian Motion Stochastic Recursive Gradient Norm Based Generalization Bound Trajectory Generalization

June 6, 2022

Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guarantees
Haotian Ju, Dongyue Li, Hongyang R. Zhang
Deep Neural Network Generalization Bound Fine Tuned Model Generalization Gap Generalization Property Generalization Guarantee Robust Fine Tuning

May 23, 2022

Generalization Gap in Amortized Inference
Mingtian Zhang, Peter Hayes, David Barber
Strong Generalization Variational Auto Generalization Gap Likelihood Based Amortized Inference

May 19, 2022

Dataset Pruning: Reducing Training Data by Examining Generalization Influence
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, Ping Li
Deep Learning Training Data Generalization Gap Training Datasets Generalization Behavior Dataset Pruning

May 13, 2022

Formal limitations of sample-wise information-theoretic generalization bounds
Hrayr Harutyunyan, Greg Ver Steeg, Aram Galstyan
Information Theoretic Generalization Gap Finite Sample Information Theoretic Generalization Bound

April 28, 2022

Formulating Robustness Against Unforeseen Attacks
Sihui Dai, Saeed Mahloujifar, Prateek Mittal
Native Robustness Adversarial Example Adversarial Training Generalization Gap Adversarial Risk

March 23, 2022

Predicting the generalization gap in neural networks using topological data analysis
Rubén Ballester, Xavier Arnal Clemente, Carles Casacuberta, Meysam Madadi, Ciprian A. Corneanu, Sergio Escalera
Neural Network Topological Data Analysis Generalization Gap Generalization Capacity

March 22, 2022

Improving Generalization in Federated Learning by Seeking Flat Minima
Debora Caldarola, Barbara Caputo, Marco Ciccone
Strong Generalization Sharpness Aware Minimization Generalization Gap Flat Minimum Generalization Capacity

March 18, 2022

Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentation
An Xu, Wenqi Li, Pengfei Guo, Dong Yang, Holger Roth, Ali Hatamizadeh, Can Zhao, Daguang Xu, Heng Huang, Ziyue Xu
Deep Learning Medical Image Segmentation Medical Image Segmentation Task Generalization Gap Federated Medical Image Segmentation

February 15, 2022

Information-Theoretic Analysis of Minimax Excess Risk
Hassan Hafez-Kolahi, Behrad Moniri, Shohreh Kasaei
Information Theoretic Generalization Gap Zero Sum Game Statistical Learning Theory Excess Risk