PAC-Bayesian generalization bounds are among the strongest distribution-dependent guarantees in statistical learning and are typically expressed in terms of the Kullback-Leibler (KL) divergence between a prior and posterior distribution. While analytically convenient, the KL divergence requires the posterior to be absolutely continuous w.r.t. the prior, imposing support constraints that can substantially restrict the choice of posterior. Moreover, the KL divergence does not account for the underlying geometry of the probability space. In this work, we derive new PAC-Bayesian bounds replacing the KL divergence with general optimal transport (OT) distances, including the total variation and Wasserstein-p distances.
Figures & tables
Figure 1: In this linear regression experiment, we compare PAC-Bayesian bounds whose remainder terms are based on different statistical distances. We compare the behavior of the bounds for different priors, posteriors and sample sizes. For each experiment, the picture on the left shows the behavior of the distance and the right shows the full bound.
Figure 2: In this experiment, we compare PAC-Bayesian bounds whose complexity terms are based on different statistical distance. We compare the behavior of the bounds for different posteriors. The picture on the left shows the behavior of the distance and the right shows the full bound.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: The posterior places a fraction κ of its mass outside the support of the prior. The Wasserstein distance remains finite and varies continuously with κ , whereas the KL divergence is infinite for every κ>0 .
Figure 4: The posterior variance σ2 decreases toward zero while the prior remains fixed. The KL divergence diverges as the posterior collapses, whereas the Wasserstein distance converges to a finite value.
Figure 5: The posterior mean is shifted by μ while the prior and posterior covariances remain fixed. The KL divergence grows quadratically in ∣μ∣2 , whereas the W2 distance grows linearly.
Figure 6: In this experiment, we compare PAC-Bayesian bounds whose complexity terms are based on different statistical distance. We compare the behavior of the bounds for different priors and posteriors.
Artificial Intelligence Research Laboratory Department of Informatics and Intelligent Systems The Pennsylvania State University · Artificial Intelligence Research Laboratory Department of Computer Science and Engineering The Pennsylvania State University
Department of Mathematics, Informatics and Geoscience, University of Trieste, Via Valerio 12/1, 34127 Trieste, Italy · 2McGovern Institute, MIT, Main Street, Cambridge, MA 02139, USA