Information-Theoretic Generalization Bounds

Follow this topic Ask about this topic

These bounds connect the training–test gap to information shared between a sample and a learned model, using mutual information, conditional mutual information, max-information, relative entropy, and rate-distortion; privacy and PAC-Bayes provide related approaches. Common techniques include supersample and selector arguments, entropy-flow and stability analyses, concentration, and algorithm-specific noise or compression, with applications to SGD, Markov learners, sequential decisions, generative models, and language-model post-training. Recent work considers settings where data dependence interacts with sequential data, continual learning, augmentation, sparse attention, or communication limits in expert routing. A central challenge is making bounds both informative and broadly applicable: general information bounds can be loose, while sharper results often require assumptions about losses, tails, mixing, variance, or algorithm structure, and information measures do not always track actual generalization.

Latest papers

22 papers6 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Generalization Bounds via Vector-Valued $L_p$ Correlation $V_α$
Oct 21, 2026Mohammad Hossein Momeni Hamaneh, Mohammad Mahdi Mojahedian, Mohammad Hossein YassaeeGeneralization Bounds Information-Theoretic Generalization Bounds
Information Systems and Security Laboratory (ISSL), Sharif University of Technology, Tehran, Iran
Learning by Compression: Risk Estimates Attain the Law-of-Large-Numbers Rate
Oct 9, 2026Simone Garatti, Marco C. CampiSample Compression Statistical Learning Theory
Dipartimento di Elettronica, Informazione e Bioingegneria - Politecnico di Milano, piazza Leonardo da Vinci 32, 20133 Milano, Italia. · Dipartimento di Ingegneria dell’Informazione - Universit`a di Brescia, via Branze 38, 25123 Brescia, Italia.
An Accuracy-Information Tradeoff for Loss-Difference Conditional Mutual Information
Oct 6, 2026Hazar YuekselStatistical Learning Theory Information-Theoretic Generalization Bounds
The optimal information complexity of VC learning
Oct 6, 2026Steve Hanneke, Juexiao WangStatistical Learning Theory Information-Theoretic Generalization Bounds
Department of Computer Science, Purdue University
On the Trade-off Between Information Loss and Generalization in Sparse Attention
Oct 3, 2026Zhongqi Fan, Zheng TanSparse Attention Generalization Bounds
Xi’an Jiaotong University
Information-Theoretic Analysis of Next-Token Prediction under Markovian Data
Sep 28, 2026Masoud Kavian, Abdellatif Zaidi, Milad SefidgaranInformation-Theoretic Generalization Bounds Next-Token Prediction
Mathematical and Algorithmic Science Laboratory, Huawei Paris Research Center, 92100 Boulogne-Billancourt, France · Laboratoire d’Informatique Gaspard Monge, Université Gustave Eiffel, 77420 Champs-sur-Marne, France
Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning
Aug 12, 2026Tieliang Gong, Zhongbo Zhang, Wen Wen +1Information-Theoretic Generalization Bounds Replay-Based Continual Learning
Tieliang Gong, Zhongbo Zhang and Wen Wen are with the School of Computer Science and Technology, Xi’an Jiaotong University, China · Yong-Jin Liu is with Department of Computer Science and Technology, Tsinghua University, China
From Privacy to Generalization: Linear Max-Information Bounds for Differentially Private Learning Algorithms
May 25, 2026Christoph H. Lampert, Max Cairney-Leeming, Hossein ZakeriniaPAC-Bayesian Generalization Bounds Differentially Private Stochastic Gradient Descent
Institute of Science and Technology Austria (ISTA) Klosterneuburg, Austria
Bounded-Rationality, Hedging, and Generalization
May 14, 2026Pedro A. OrtegaInformation-Theoretic Generalization Bounds Bounded Rationality
Daios Technologies
On the Generalization of Knowledge Distillation: An Information-Theoretic View
May 13, 2026Bingying Li, Haiyun HeInformation-Theoretic Generalization Bounds Statistical Learning Theory
Internet of Things Thrust, Information Hub The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China
Information-Theoretic Generalization Bounds for Sequential Decision Making
May 12, 2026Futoshi Futami, Masahiro FujisawaInformation-Theoretic Generalization Bounds Generalization Bounds
The University of Osaka / RIKEN AIP / The University of Tokyo · The University of Osaka / RIKEN AIP
Information-theoretic Limits of Learning and Estimation
May 6, 2026Abbas El Gamal, Maxim RaginskyInformation-Theoretic Generalization Bounds Minimax Estimation
Stanford University* · UIUC**
Expert Routing for Communication-Efficient MoE via Finite Expert Banks
May 6, 2026Mohammad Reza Deylam Salehi, Ali KhalesiMixture of Experts Information-Theoretic Generalization Bounds
IEEE Graduate Member · Nice, France · Institut Polytechnique des Sciences Avancées (IPSA) +1
Information-Theoretic Generalization Bounds for Stochastic Gradient Descent with Predictable Virtual Noise
Apr 30, 2026Mohammad PartohaghighiInformation-Theoretic Generalization Bounds Stochastic Gradient Descent
Electrical Engineering and Computer Science University of California, Merced Merced, CA, USA
Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewards
Apr 12, 2026Huiming Zhang, Binghan Li, Wan Tian +1Information-Theoretic Generalization Bounds LLM Alignment
BUAA and BAIC-FBPC · BUAA · PKU +1
Conjugate Learning Theory: Uncovering the Mechanisms of Trainability and Generalization in Deep Neural Networks
Feb 18, 2026Binchuan QiNeural Network Generalization Neural Network Optimization
Tongji University, Siping Street, 200092 Shanghai, China · Zhejiang Yuying College of Vocational Technology, Sihao Street, Hangzhou, 310018, Zhejiang, China
Mixture-of-Experts under Finite-Rate Gating: Communication--Generalization Trade-offs
Feb 16, 2026Ali Khalesi, Mohammad Reza Deylam SalehiMixture-of-Experts Models Communication-Efficient Mixture-of-Experts
Institut Polytechnique des Sciences Avancées (IPSA) and LINCS Lab, Paris, France · IEEE member, Nice, France
The geometry of invariant learning: an information-theoretic analysis of data augmentation and generalization
Feb 16, 2026Abdelali Bouyahia, Frédéric LeBlanc, Mario MarchandData Augmentation Generalization Bounds
Département d’informatique et génie logiciel Université Laval Québec, QC, Canada · Institut intelligence et données Université Laval Québec, G1V 0A6, Canada
Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Training
Jul 2, 2025Ismail Labiad, Mathurin Videau, Matthieu Kowalski +4Privacy-Preserving Language Models Evolutionary Optimization
Meta FAIR · Université Paris-Saclay · LISN, Inria, CNRS +2
Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis
Jun 1, 2025Qi Chen, Jierui Zhu, Florian ShkurtiVariational Autoencoders Neural Network Generalization
Department of Computer Science, University of Toronto · Data Science Institute · Vector Institute +2
Generalization Bounds for Markov Algorithms through Entropy Flow Computations
Feb 11, 2025Benjamin Dupuis, Maxime Haddouche, George Deligiannidis +1Information-Theoretic Generalization Bounds Generalization Bounds
INRIA - D´epartement d’Informatique de l’Ecole Normale Sup´erieure PSL Research University - CNRS Paris, France · Department of Statistics, University of Oxford Oxford, UK
The Method of Gaps: Exact Expressions for the Generalization Error of Supervised Learning Algorithms
Nov 18, 2024Samir M. Perlaza, Xinying ZouInformation-Theoretic Generalization Bounds Statistical Learning Theory
INRIA, Centre Inria d’Universit´e Cˆote d’Azur, 06902 Sophia Antipolis, France

Oct 21, 2026J/K move · Enter open · S save

Generalization Bounds via Vector-Valued $L_p$ Correlation $V_α$

Mohammad Hossein Momeni Hamaneh, Mohammad Mahdi Mojahedian, Mohammad Hossein Yassaee

Information Systems and Security Laboratory (ISSL), Sharif University of Technology, Tehran, Iran

Generalization Bounds Information-Theoretic Generalization Bounds

Open paper AskPDF arXiv

Understanding generalization of randomized learning algorithms remains a central challenge. Mutual-information and total-variation bounds provide complementary ways to quantify data--hypothesis dependence, but they can behave quite differently in high-dimensional regimes. We study a one-parameter family of dependence measures, the vector-valued

L_p

-norm correlation

V_α

, and derive generalization bounds whose dependence terms interpolate between total variation (TV) at

α=1

and variance and

χ^2

related behavior at

α=2

. For

α>2

, the Hölder-conjugate loss moment is below two, yielding heavy-tailed specializations that can remain finite even when the loss variance is infinite. A particularly interpretable specialization occurs at

α=2

, where the bound can be expressed in terms of the variability of the algorithm's output density across datasets,

\int \sqrt{\mathrm{Var}_S[p(w\mid S)]}~ dw

. We show that the entire

V_α

family is invariant under smooth bijective reparameterizations of the hypothesis space. We also introduce an adaptive density stability (ADS) condition that implies explicit control of

V_2

, and we verify it for Gibbs-type posteriors under bounded losses (yielding rates that depend on the inverse temperature and sample size). To address scaling, we extend the framework to a conditional formulation using supersamples, analogous in spirit to conditional mutual information (CMI), and discuss evaluations in the output space as a practical surrogate for estimating dependence. Simulations on Bayesian linear regression illustrate how tuning

α

can change the tightness of the resulting bounds across regimes. Additional supersample experiments on noisy MNIST and on a pretrained ResNet-50 for CIFAR-10 show similar behavior of the projected loss-space bounds beyond the original clean-MNIST setting.