Non Vacuous Generalization Bound

Non-vacuous generalization bounds aim to provide mathematically rigorous guarantees that a machine learning model's performance on unseen data will be similar to its training performance, avoiding overly loose bounds that are practically useless. Current research focuses on developing such bounds for large language models (LLMs), neural networks (including convolutional and shallow architectures), and federated learning settings, employing techniques like compression, PAC-Bayesian analysis, and Rademacher complexity. Achieving non-vacuous bounds for complex models like LLMs is a significant challenge, with recent progress leveraging novel compression methods and exploiting the inherent structure of these models to obtain tighter, more informative bounds. This work contributes to a deeper understanding of generalization in deep learning and informs the design of more reliable and trustworthy AI systems.

Papers

October 9, 2024

Unifying and Verifying Mechanistic Interpretations: A Case Study with Group Operations
Wilson Wu, Louis Jaburi, Jacob Drori, Jason Gross
Neural Network Case Study Mechanistic Interpretability Anti Unification Economic Operation Model Understanding Non Vacuous Generalization Bound

July 25, 2024

Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models
Sanae Lotfi, Yilun Kuang, Brandon Amos, Micah Goldblum, Marc Finzi, Andrew Gordon Wilson
Large Language Model Generalization Bound K TOKEN Larger Language Model Field Data Tighter Bound Non Vacuous Generalization Bound

July 9, 2024

A Generalization Bound for Nearly-Linear Networks
Eugene Golikov
Neural Network Generalization Bound Generalization Error Bound Linear Network Non Vacuous Generalization Bound

December 28, 2023

Non-Vacuous Generalization Bounds for Large Language Models
Sanae Lotfi, Marc Finzi, Yilun Kuang, Tim G. J. Rudner, Micah Goldblum, Andrew Gordon Wilson
Large Language Model Language Model Generalization Bound Non Vacuous Generalization Bound

October 17, 2023

Federated Learning with Nonvacuous Generalisation Bounds
Pierre Jobic, Maxime Haddouche, Benjamin Guedj
Generalization Bound PAC Bayesian Private Prediction Non Vacuous Generalization Bound Global Posterior

July 15, 2023

Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networks
Dominik Schnaus, Jongseok Lee, Daniel Cremers, Rudolph Triebel
Neural Network Strong Generalization Uncertainty Estimation Generalization Bound Generalization Guarantee Expressive Neural Non Vacuous Generalization Bound Learned Prior

February 17, 2023

PAC-Bayesian Generalization Bounds for Adversarial Generative Models
Sokhna Diarra Mbacke, Florence Clerc, Pascal Germain
Generative Model Generalization Bound Wasserstein Gan PAC Bayesian Generalization Bound Non Vacuous Generalization Bound

November 24, 2022

PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization
Sanae Lotfi, Marc Finzi, Sanyam Kapoor, Andres Potapczynski, Micah Goldblum, Andrew Gordon Wilson
Deep Learning Deep Neural Network Strong Generalization Generalization Bound Upper Bound PAC Bayes Bound Non Vacuous Generalization Bound

August 8, 2022

On Rademacher Complexity-based Generalization Bounds for Deep Learning
Lan V. Truong
Deep Learning Convolutional Neural Network Generalization Error Bound Rademacher Complexity Non Vacuous Generalization Bound Lipschitz Activation

February 3, 2022

Non-Vacuous Generalisation Bounds for Shallow Neural Networks
Felix Biggs, Benjamin Guedj
PAC Bayesian Shallow Neural Network Sigmoid Function Vanilla Gradient Descent Non Vacuous Generalization Bound