Generalization Guarantee

Generalization guarantees in machine learning aim to provide theoretical bounds on how well a model trained on a specific dataset will perform on unseen data. Current research focuses on developing tighter bounds for various model architectures, including deep neural networks, and algorithms like gradient descent and direct preference optimization, often leveraging techniques from information theory, stability analysis, and PAC-Bayes theory. These efforts are crucial for building reliable and trustworthy machine learning systems, particularly in safety-critical applications where performance on unseen data is paramount, and for understanding the relationship between model complexity, training data, and generalization ability. Improved generalization guarantees can lead to more efficient algorithms and more robust model deployment.

Papers

October 20, 2024

On Cold Posteriors of Probabilistic Neural Networks: Understanding the Cold Posterior Effect and A New Way to Learn Cold Posteriors with Tight Generalization Guarantees
Yijie Zhang
Bayesian Inference Bayesian Approach PAC Bayesian Bayesian Deep Learning Generalization Guarantee Bayesian Model Probabilistic Neural Network Cold Posterior

October 17, 2024

Sample Compression Hypernetworks: From Generalization Bounds to Meta-Learning
Benjamin Leblanc, Mathieu Bazinet, Nathaniel D'Amours, Alexandre Drouin, Pascal Germain
Generalization Bound Generalization Guarantee Sample Compression

October 11, 2024

Online-to-PAC generalization bounds under graph-mixing dependencies
Baptiste Abélès, Eugenio Clerico, Gergely Neu
Generalization Bound Statistical Learning Generalization Guarantee Question Based Dependency PAC Bayesian Generalization Bound

September 26, 2024

Sample Compression Unleashed: New Generalization Bounds for Real Valued Losses
Mathieu Bazinet, Valentina Zantedeschi, Pascal Germain
Generalization Bound Generalization Guarantee Planning Loss Sample Compression One v the Rest Loss

September 13, 2024

Optimization and Generalization Guarantees for Weight Normalization
Pedro Cisneros-Velarde, Zhijie Chen, Sanmi Koyejo, Arindam Banerjee
Optimization Purpose Generalization Guarantee Smooth Activation Function Training Convergence Weight Normalization

August 28, 2024

Does Data-Efficient Generalization Exacerbate Bias in Foundation Models?
Dilermando Queiroz, Anderson Carlos, Maíra Fatoretto, André Anjos, Lilian Berton, Luis Filipe Nakayama
Foundation Model Robust Model Generalization Guarantee Label Efficiency

August 20, 2024

An Information-Theoretic Approach to Generalization Theory
Borja Rodríguez-Gálvez, Ragnar Thobaben, Mikael Skoglund
Distribution Generalization Information Theoretic Generalization Guarantee PAC Bayesian Bound Information Theoretic Approach Generalization Theory

August 6, 2024

On the Generalization of Preference Learning with DPO
Shawn Im, Yixuan Li
Large Language Model Strong Generalization Direct Preference Optimization Preference Learning Generalization Guarantee Generalization Theory

June 11, 2024

Loss Gradient Gaussian Width based Generalization and Optimization Guarantees
Arindam Banerjee, Qiaobo Li, Yingxue Zhou
Strong Generalization Hessian Matrix Generalization Guarantee Rademacher Complexity Loss Gradient Optimality Guarantee Gradient Dominance Property

May 22, 2024

Adaptive Data Analysis for Growing Data
Neil G. Marchant, Benjamin I. P. Rubinstein
Generalization Bound Generalization Guarantee Growth Monitoring Optimal Data Adaptive Query

April 22, 2024

Data-Driven Performance Guarantees for Classical and Learned Optimizers
Rajiv Sambharya, Bartolomeo Stellato
Generalization Guarantee Performance Guarantee Learned Optimizers Classical Optimization Parametric Optimization

April 14, 2024

Generalization Error Bounds for Learning under Censored Feedback
Yifan Yang, Ali Payani, Parinaz Naghizadeh
Generalization Error Error Bound Generalization Error Bound Generalization Guarantee

April 4, 2024

Learning-to-Optimize with PAC-Bayesian Guarantees: Theoretical Considerations and Practical Implementation
Michael Sucker, Jalal Fadili, Peter Ochs
Practical Implementation PAC Bayesian Generalization Guarantee Learned Optimizers PAC Bayesian Bound Theoretical Consideration

February 5, 2024

Minimum Description Length and Generalization Guarantees for Representation Learning
Milad Sefidgaran, Abdellatif Zaidi, Piotr Krasnowski
Representation Learning Generalization Bound Generalization Guarantee Minimum Description Length

July 18, 2023

Batched Predictors Generalize within Distribution
Andreas Loukas, Pan Kessel
Product Distribution Generalization Property Generalization Guarantee Easy to Hard Generalization

July 15, 2023

Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networks
Dominik Schnaus, Jongseok Lee, Daniel Cremers, Rudolph Triebel
Neural Network Strong Generalization Uncertainty Estimation Generalization Bound Generalization Guarantee Expressive Neural Non Vacuous Generalization Bound Learned Prior

July 12, 2023

Tackling Combinatorial Distribution Shift: A Matrix Completion Perspective
Max Simchowitz, Abhishek Gupta, Kaiqing Zhang
Matrix Completion Singular Value Decomposition Generalization Guarantee Training Distribution Cartesian Product Extrapolation

July 4, 2023

Generalization Guarantees via Algorithm-dependent Rademacher Complexity
Sarah Sachs, Tim van Erven, Liam Hodgkinson, Rajiv Khanna, Umut Simsekli
Generalization Bound Generalization Guarantee Rademacher Complexity

June 15, 2023

On Certified Generalization in Structured Prediction
Bastian Boll, Christoph Schnörr
Generative Model Generalization Bound Structured Prediction Generalization Guarantee

June 8, 2023

Federated Learning under Covariate Shifts with Generalization Guarantees
Ali Ramezani-Kebrya, Fanghui Liu, Thomas Pethick, Grigorios Chrysos, Volkan Cevher
Empirical Risk Minimization Covariate Shift Communication Efficient Generalization Guarantee