Generalization Bound

Generalization bounds in machine learning aim to quantify a model's ability to perform well on unseen data, based on its performance on training data. Current research focuses on developing tighter bounds for various architectures, including neural networks (especially deep and "nearly-linear" networks), large language models, and graph neural networks, often employing techniques like sample compression, PAC-Bayesian analysis, and information-theoretic approaches. These advancements are crucial for understanding and improving the reliability and robustness of machine learning models, particularly in high-stakes applications where generalization is paramount. The development of practically computable and informative bounds remains a significant challenge and active area of investigation.

Papers

November 8, 2024

Generalization, Expressivity, and Universality of Graph Neural Networks on Attributed Graphs
Levi Rauchwerger, Stefanie Jegelka, Ron Levie
Graph Neural Network Strong Generalization Generalization Bound Gene Level GNN Universal Approximation K$ Universality Behavior Expressivity Style Attributed Graph

November 5, 2024

A Convex Relaxation Approach to Generalization Analysis for Parallel Positively Homogeneous Networks
Uday Kiran Reddy Tadipatri, Benjamin D. Haeffele, Joshua Agterberg, René Vidal
Empirical Risk Minimization Generalization Bound Convex Relaxation Homogeneous Neural Network Two Layer Network Homogeneous Diffusive Network

October 30, 2024

Generalization Bounds via Conditional $f$-Information
Ziqiao Wang, Yongyi Mao
Mutual Information Generalization Bound Conditional Reasoning Conditional Mutual Information Information Theoretic Generalization Bound Unbounded Loss

October 21, 2024

On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Bounds
Matteo Vilucchio, Nikolaos Tsilivis, Bruno Loureiro, Julia Kempe
Adversarial Attack Adversarial Training High Dimensional Generalization Bound Geometric Analysis Gradient Norm

October 17, 2024

Sample Compression Hypernetworks: From Generalization Bounds to Meta-Learning
Benjamin Leblanc, Mathieu Bazinet, Nathaniel D'Amours, Alexandre Drouin, Pascal Germain
Generalization Bound Generalization Guarantee Sample Compression

October 15, 2024

On Rank-Dependent Generalisation Error Bounds for Transformers
Lan V. Truong
Transformer Megatron Decepticons Generalization Bound Generalization Error Efficient Transformer Low Rank Matrix Generalization Error Bound

October 14, 2024

Learning via Surrogate PAC-Bayes
Antoine Picard-Weibel, Roman Moscoviz, Benjamin Guedj (UCL, UCL-CS, Inria, Inria-London, MODAL)
LeArning Abstract Generalization Bound PAC Bayesian PAC Bayes Bound

October 11, 2024

October 10, 2024

Generalization Bounds and Model Complexity for Kolmogorov-Arnold Networks
Xianyang Zhang, Huijuan Zhou
Loss Function Multi Layer Generalization Bound Kolmogorov Arnold Network Reproducing Kernel Hilbert Space Model Complexity

October 2, 2024

Which Algorithms Have Tight Generalization Bounds?
Michael Gastpar, Ido Nachum, Jonathan Shafer, Thomas Weinberger
Practical Algorithm Inductive Bias Generalization Bound

September 26, 2024

Sample Compression Unleashed: New Generalization Bounds for Real Valued Losses
Mathieu Bazinet, Valentina Zantedeschi, Pascal Germain
Generalization Bound Generalization Guarantee Planning Loss Sample Compression One v the Rest Loss

July 29, 2024

Generalization bounds for regression and classification on adaptive covering input domains
Wen-Liang Hwang
Neural Network Strong Generalization Classification Code Novel Regression Generalization Bound Generalization Error Regression Task Source Domain

July 25, 2024

Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models
Sanae Lotfi, Yilun Kuang, Brandon Amos, Micah Goldblum, Marc Finzi, Andrew Gordon Wilson
Large Language Model Generalization Bound K TOKEN Larger Language Model Field Data Tighter Bound Non Vacuous Generalization Bound

July 24, 2024

Generalization Bounds of Surrogate Policies for Combinatorial Optimization Problems
Pierre-Cyril Aubin-Frankowski, Yohann De Castro, Axel Parmentier, Alessandro Rudi
Generalization Bound Combinatorial Optimization Problem Contextual Bayesian Optimization Stochastic Scheduling Surrogate Safety Measure

July 9, 2024

A Generalization Bound for Nearly-Linear Networks
Eugene Golikov
Neural Network Generalization Bound Generalization Error Bound Linear Network Non Vacuous Generalization Bound

June 20, 2024

Optimal deep learning of holomorphic operators between Banach spaces
Ben Adcock, Nick Dexter, Sebastian Moraga
Deep Learning Generalization Bound Operator Learning Hilbert Space Banach Space

June 6, 2024

Slicing Mutual Information Generalization Bounds for Neural Networks
Kimia Nadjahi, Kristjan Greenewald, Rickard Brüel Gabrielsson, Justin Solomon
Neural Network Generalization Bound Information Theoretic Generalization Bound Random Subspace

June 5, 2024

Unified PAC-Bayesian Study of Pessimism for Offline Policy Learning with Regularized Importance Sampling
Imad Aouali, Victor-Emmanuel Brunel, David Rohde, Anna Korba
Generalization Bound Unified Alignment Importance Sampling PAC Bayesian Offline Policy Learning Weight Regularization

May 30, 2024

Length independent generalization bounds for deep SSM architectures
Dániel Rácz, Mihály Petreczky, Bálint Daróczy
State Space Model Generalization Bound Input Sequence Sequence Generation Stability Constraint Crown Rump Length Sequential Generation