Pac-Bayesian Theory

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 6Week of Sep 21

Latest papers 36

Sep 15, 2026stat.ML

On the disintegration of the stochastic majority vote: From PAC-Bayesian bounds to a self-bounding algorithm

Weighted majority votes are central to many successful ensemble methods. PAC-Bayesian theory provides tight generalization guarantees for such models by analyzing the expected risk of stochastic classifiers, while analyzing the risk of deterministic majority votes relies on surrogate bounds. To avoid these surrogates, Zantedeschi et al. ( 2021) introduced guarantees for stochastic majority votes, but the resulting models remain randomized. In this paper, we propose a derandomization framework for stochastic majority votes. To do so, we apply recent advances in disintegrated PAC-Bayesian theory directly to the space of majority vote weight vectors, transforming stochastic guarantees into certificates for a single deterministic majority vote. We derive two families of high-probability generalization bounds, covering both data-independent and data-dependent constructions of the ensemble, which naturally lead to a self-bounding learning algorithm optimizing deterministic majority vote guarantees.

Julien Bastian, Benjamin Leblanc, Pascal Germain +4

LabHC · Universit´e Jean Monnet Saint-´Etienne, CNRS, Institut d Optique Graduate School, Laboratoire Hubert Curien UMR 5516, F-42023, Saint-Etienne, France · LabHC, UJM, MALICE +6

Majority Voting Generalization Bounds Pac-Bayesian Theory

Sep 14, 2026cs.LG

Quantifying the Value of Privileged Information Using a PAC-Bayesian Approach

In practice, various learning scenarios provide access to auxiliary features exclusively during training. Incorporating such data to enhance model performance gave rise to a paradigm known as Learning Using Privileged Information (LUPI). While this extra information is intended to improve the resulting model, establishing a generalized, cohesive understanding of how privileged information (PI) transfers useful knowledge remains a challenge. Vapnik's original theory and subsequent works offer performance guarantees in certain cases, but these results are inherently per-algorithm and rely on setting-specific proof approaches. Consequently, a more general framework explaining how and when PI transfers useful knowledge is still missing. To bridge this gap, we introduce an algorithm-agnostic, information-theoretic approach based on the PAC-Bayes framework. Rather than asking whether a particular algorithm exploits PI, we ask how much value it could offer: comparing the tightest achievable risk bound with and without PI yields its potential - an upper limit on the extractable gain. We introduce a metric that quantifies this potential directly from empirical training risk, bypassing the need for test-time data access, and validate our findings in both supervised and unsupervised settings. The results demonstrate a robust correspondence between our training-time metric and true test-time performance gains. Ultimately, this work takes a necessary step toward an information-theoretic understanding of LUPI, and quantifying the potential of privileged features before committing to a model.

Vasily Bokov (aQa, Leiden University, The Netherlands +24

Privacy-Preserving Machine Learning Pac-Bayesian Theory Information-Theoretic Limits Empirical Risk Minimization Value

Sep 12, 2026cs.LG

Prevalence Determines Precision:Silent Contamination in Detector-Defined Datasets

Many ML datasets are constructed by running a detector, heuristic, or model over candidate pools; accepted items become labels. Dataset precision is then governed by true-positive prevalence in each pool via Bayes, not solely by detector quality. Using one instrument and period, we hold a detector-defined event dataset plus an independent official index labeling every detected item as real or phantom. One detector, three pools yield phantom rates 81.7%, 9.0%, and 0.0%. Transferring precision from the two high-rate pools to the low-rate pool predicts 0.955 versus measured 0.183, a +422% error; the Bayes expression predicts all three within 3.3%. The detected response curve is an exact convex combination of a true-event and a phantom component (residual 1.1e-16), with phantoms outnumbering true events 473 to 308, so contamination is a second signal with detector-inherited shape, not additive noise. Contamination direction depends on the estimator: on identical windows one statistic is diluted and another inflated because its denominator is also contaminated. A common normalization turns the estimator into a mean of ratios whose expectation need not exist; on the same 335 events it returns 0.40 where the well-defined estimator returns 0.10.

Jia Huang, Yankai Wan, Yangjun Ou

Guanghua School of Management, Peking University · College of Artificial Intelligence, Jilin University · School of Mathematical Sciences, Peking University

Prevalence False-Negative Rate Detection Performance Pac-Bayesian Theory

Aug 11, 2026cs.LG

PAC-Bayes Beyond Parameter Space: Behavioral Equivalence, Z-Information, and Exact Complexity Decomposition

PAC-Bayes theory provides generalization guarantees by controlling the Kullback--Leibler (KL) divergence between posterior and prior distributions over a chosen hypothesis representation. However, predictive risk depends only on the predictive behavior induced by a hypothesis, not on the particular internal realization that implements that behavior. In over-parameterized systems, many distinct configurations induce identical predictive behavior, yet the classical PAC-Bayes KL divergence does not distinguish uncertainty over predictive behavior from variation among behaviorally equivalent realizations. We show that this distinction induces an exact structural decomposition of classical PAC-Bayes complexity. We formalize behavioral equivalence through a measurable behavior map and use measure disintegration to decompose probability measures on the configuration space into a distribution over predictive behaviors and conditional distributions over behavioral fibers. This yields an exact decomposition of the classical PAC-Bayes KL divergence into a behavior-selection term and a realization-level term given by an expected conditional KL within fibers. We define Z-information as the negative of this realization-level contribution: the exact gap between the KL divergence and the complexity of uncertainty over predictive behavior alone. We further show that the behavior-selection term admits an exact variational characterization: it is the minimum KL divergence among all posteriors inducing the same distribution over predictive behaviors, attained by a canonical fiber-symmetrized representative. Finally, we show that symmetry, behavior-preserving directions, fiber geometry, and invariance under fiber-preserving perturbations arise naturally from the same behavior-map structure. Together, these results identify predictive behavior as the natural object of PAC-Bayes complexity.

Vasant G. Honavar, Satish Kumar Keshri, Neil Ashtekar +1

Artificial Intelligence Research Laboratory Department of Informatics and Intelligent Systems The Pennsylvania State University · Artificial Intelligence Research Laboratory Department of Computer Science and Engineering The Pennsylvania State University

Pac-Bayesian Theory Kullback-Leibler Divergence Variational Equivalence

Aug 4, 2026cs.LG

Sample Complexity of Multicalibration for Multilevel Properties

Calibration requires a predictor to be unbiased after conditioning on its own predictions. Multicalibration asks for this guarantee simultaneously across a collection of groups. Many prediction tasks ask for several related features of the same conditional outcome distribution: variance is defined relative to the mean, skewness relative to both mean and variance, and conditional value at risk relative to a quantile. We study multicalibration for a sequence of

k

properties in which each property is identifiable once the preceding properties are fixed. This framework includes Bayes pairs but does not require the properties to arise from a single loss. For every fixed

k\ge2

, we establish matching upper and lower sample-complexity bounds up to logarithmic factors under regularity conditions. Even with only polylogarithmically many binary groups, achieving multicalibration error

\varepsilon

requires

\widetildeΩ(\varepsilon^{-(k+2)})

samples. Conversely, for any finite group family

\mathcal G

, we give a randomized learner using

O(\varepsilon^{-(k+2)}+\varepsilon^{-2}\log|\mathcal G|)

samples. Thus the sample complexity is

\widetildeΘ(\varepsilon^{-(k+2)})

for polynomial-size group families. We instantiate the theory for three canonical examples.

Jiuyao Lu, Krishnakumar Balasubramanian, Aleksandr Podkopaev +1

Department of Statistics and Data Science, The Wharton School, University of Pennsylvania · 2Amazon · Department of Statistics, University of California, Davis

Optimal Sample Complexity Sample Complexity Pac-Bayesian Theory Finite Groups

Aug 1, 2026cs.AI

BayesSeg: A Bayesian Optimization Framework for State Segmentation of Electricity Consumption Time Series

In Non-Intrusive Load Monitoring (NILM), adaptive segmentation of electricity consumption time series is critical for appliance recognition. However, prevailing methods face challenges including heuristic parameter tuning, boundary sensitivity, and metric saturation. This paper proposes BayesSeg, a unified framework integrating time-series segmentation, multidimensional evaluation, and automatic parameter optimization. The segmentation layer employs a dual steady-state criterion based on the tail value and mean of preceding subsequences, combined with a sequential extraction and complement-set parsing strategy, to achieve precise unsupervised partitioning of steady-state and transition-state segments. The evaluation layer maps segmentation results to binary state sequences and formulates a composite metric integrating an event-level F1 score (event_F1) with Normalized Mutual Information (NMI). The event_F1 quantifies switching-event precision and recall via tolerance matching, while NMI captures global structural consistency, jointly overcoming the boundary sensitivity and limited discriminability of point-wise metrics. In the optimization layer, the composite score serves as the objective function for Bayesian optimization, which constructs a TPE surrogate model for efficient global parameter-space exploration. Experiments on the SustDataED2 dataset demonstrate that Bayesian optimization requires only ~100 objective evaluations to locate a parameter region within 0.35% deviation of the exhaustive grid-search optimum. The framework achieves a weighted composite score of 0.7149 and an event_F1 of 0.9340 while reducing optimization latency from ~5300 seconds to under 1 second, a speedup exceeding 5700x. BayesSeg automates segmentation configuration and provides a scalable, efficient solution for time-series analysis in NILM and related domains.

Zhenya Zhang, Wendi Zhu, Ping Wang +2

Anhui Province Key Laboratory of Intelligent Building & Building Energy Saving, Anhui Jianzhu University, Hefei 230022, China · School of Electronic and Information Engineering, Anhui Jianzhu University, Hefei 230022, China · School of Economics and Management, Anhui Jianzhu University, Hefei 230022, China +1

Bayesian Optimization Pac-Bayesian Theory

Jul 29, 2026cs.LG

BayesAME: Bayesian Active Model Evaluation

Evaluating large generative models across benchmarks is time-consuming and computationally expensive. This drives the need for methods that can estimate full benchmark performance by evaluating models on only a subset of items, known as a coreset. Current literature mostly requires the practitioner to input a coreset size. However, when reliable performance estimation takes priority over efficiency, an evaluation method should also be capable of automatically determining a coreset size that reflects this priority. We introduce BayesAME, a sequential Bayesian framework specifically targeting automatic determination of the coreset size. BayesAME models performance as a random variable by defining a latent ability for each group of items sharing the same historical model performances, with a joint prior distribution encoding the belief that the target model behaves similarly to these historical models. The posterior distribution over these abilities is used to derive performance estimators, quantify performance uncertainty, and select items to add to the coreset via an information-gain criterion. The coreset is iteratively augmented until the performance estimate fluctuation and the performance uncertainty fall below their respective user-defined thresholds. We propose a multi-target extension that captures performance correlations across multiple target models to further reduce the coreset size. Through extensive experiments across diverse benchmarks, we demonstrate that BayesAME consistently outperforms sequential adaptations of existing methods. Crucially, our comprehensive analysis addresses recent skepticism in the literature, establishing that non-random coreset selection is advantageous over random selection. Finally, we highlight that leveraging continuous response log-likelihoods over traditional binary scores significantly enhances estimation accuracy.

Paula Cordero Encinar, Taylan Cemgil, Arnaud Doucet +2

Imperial College London (work conducted while at Google DeepMind) · Google DeepMind

Geometry-Aware Uncertainty Coresets Model Selection Model Evaluation Pac-Bayesian Theory Generative Models Prior Knowledge Integration

Jul 24, 2026cs.LG

Generalised Balanced Softmax: A Finite-Data Perspective on Logit Adjustment for Long-Tailed Recognition

Models trained on long-tailed data using standard softmax tend to exhibit higher training error and a larger generalisation gap for classes with fewer training samples. We characterise this class-wise disparity as the preference issue and quantify it using a new metric, the model imbalance level

I

. To understand this issue, we analyse how imbalanced training data adversely affects class-wise gradients under standard softmax training. This paper then develops a finite-data Generalised Balanced Softmax (GBS) framework for analysing and mitigating the preference issue. The framework uses the training-time logit adjustment

z_{nc}+β\log|N_c|

, which is algebraically identical to the training-time logit-adjusted loss of Menon et al. (2021) when

τ=β

. The case

β=1

also coincides with Balanced Softmax and with the unit adjustment supported by the Fisher-consistency argument under the true data distribution, corresponding to an idealised infinite-data setting. Building on this existing loss family, this paper uses a heuristic power-law assumption to motivate the adjustable coefficient and studies how

β

affects trained models. Across the evaluated long-tailed benchmarks,

β=1

does not attain the highest average testing recall on most datasets, showing that a different coefficient can be preferable when training on finite data. The selected values of

β

reduce

I

and improve average testing recall relative to the

β=1

reference, while retaining negligible computational overhead and compatibility with existing representation-learning frameworks.

Yi-Hang Zhu, Rajeev Raman, Shiqi Su +4

School of Computing and Mathematical Sciences, University of Leicester, Leicester, LE1 7RH, UK · Department of Physics and Astronomy, University of Leicester, Leicester, LE1 7RH, UK · Scientific Computing, Rutherford Appleton Laboratory, Science and Technology Facilities Council, Didcot, OX11 0QX, UK +2

Imbalanced Classification Gumbel-Softmax Relaxation Long-Tailed Distribution Pac-Bayesian Theory Deep Learning Activation Functions

Jul 23, 2026stat.ML

Simulation-Based Empirical Bayes

Empirical Bayes (EB) performs simultaneous inference across many related latent variables. Classical EB assumes that the likelihood p(x | z) is tractable. In many scientific applications, however, the likelihood is available only through a simulator. This paper develops EB for such implicit likelihoods. We introduce simulation-based empirical Bayes (SBEB), which connects nonparametric EB to simulation-based inference (SBI). SBEB computes EB estimates without an explicit density by using the observed data, simulator samples, and an amortized inference network. SBEB iteratively refines the fitted EB prior toward the population prior. With several scientific simulators and real-world data, we demonstrate that SBEB improves accuracy over SBI with a fixed prior.

Xinwei Shen, Diana Cai, Cheng Zhang +1

Department of Statistics University of Washington · Department of Computer Science Cornell University · School of Mathematical Sciences and Center for Statistical Science Peking University +1

Empirical Bayes Pac-Bayesian Theory

Jul 20, 2026cs.LG

PAC--Bayes Bounds on Quotient Parameter Spaces: Geometry-induced Implicit-Bias Priors

Overparameterized models often have continuous parameter symmetries, so different parameters define the same predictor. We show that PAC--Bayesian analysis should be performed on the quotient predictor space: pushing a prior and posterior to the quotient preserves the empirical and population Gibbs risks while removing the nonnegative KL contribution caused solely by how the two distributions differ among parameterizations of the same predictor. Quotienting alone does not determine which prior to use. We construct a canonical choice of one parameterization for each predictor and account for the geometric volume of its equivalent parameterizations. This transforms a neutral reference prior into a data-independent prior that reflects the model's implicit bias. It approximates the ideal but inadmissible posterior-matched prior, which would minimize the KL term by depending on the training data. The resulting certificate is tighter exactly when this geometry-induced prior has smaller KL divergence from the learned quotient posterior than the neutral prior. We test this prediction in Fourier regression with a Hadamard parameterization and in Query-Key attention, using ordinary SGD without an explicit regularizer. The implicit-bias prior reduces the mean quotient-space KL by

40.69\%

and the mean PAC--Bayes certificate by

21.40\%

in the Fourier-Hadamard experiment. The smaller, prior-scale-dependent improvement in Query-Key attention confirms the predicted conditional nature of the effect.

Nicola Aladrah, Fabio Anselmi

Department of Mathematics, Informatics and Geoscience, University of Trieste, Via Valerio 12/1, 34127 Trieste, Italy · 2McGovern Institute, MIT, Main Street, Cambridge, MA 02139, USA

Pac-Bayesian Theory Overparameterization Symmetry

Jul 20, 2026cs.LG

The Calibration Channel Determines the Bayes-Error Proxy: An Exact Law for Temperature-Induced Distortion

The soft-label Bayes-error estimator beta(z) = E[min(z, 1-z)] of Ishida et al. estimates the irreducible error of a binary task directly from probability-valued labels. Recent work by Ushio et al. showed that this estimator is fragile when the probabilities are not the true posterior: even perfectly calibrated soft labels can yield a substantially inaccurate estimate, and they propose isotonic calibration as a consistent remedy. We complement that line of work by characterizing exactly how the most widely used post-hoc calibration map -- temperature scaling -- distorts the proxy. We prove an exact, model-free identity reducing the temperature-scaled proxy to the classifier's margin distribution, from which we obtain (i) strict monotonicity in the temperature and (ii) a continuous bijection from the temperature axis onto the open interval (0, 1/2), so that a fixed classifier -- with fixed decisions and fixed 0-1 error -- can be made to report any proxy value whatsoever. Under a Gaussian model of the logits we further derive a two-parameter closed form for the entire proxy-versus-temperature curve. Across CIFAR-10, Fashion-MNIST, and SVHN (eight binary tasks), the proxy varies by 56x to 980x at constant test error, the closed form reproduces the empirical curve to within 0.018, and the calibration temperature that minimizes the expected calibration error does not coincide with any stable proxy value. Our results give a precise, predictive account of the distortion whose existence motivates calibration-based remedies, and they reinforce the practical recommendation that a proxy value is meaningful only together with the mechanism that produced its probabilities.

Shreyas Pradeepkumar Khandale

School of Computing Binghamton University (SUNY) Binghamton, NY, USA

Pac-Bayesian Theory Calibrated Uncertainty Equal Error Rate Recalibration Exact

Jul 17, 2026cs.RO

BayesContact: Uncertain Pose Estimation via Visuo-Tactile Proposals and Simulation-based Inference

Contact-rich manipulation requires pose estimates that are often more accurate than what depth-only sensing provides. Existing methods, relying on vision and contact, employ costly offline training procedures that need to be retrained for new environments and geometries. We propose BayesContact, a Simulation-Based Inference framework for visuo-tactile pose estimation in peg-in-hole insertion. BayesContact maintains a particle belief over object pose and fuses depth observations with force/torque-derived contact evidence. We employ simulation based forward models to approximate these observation likelihoods. For each pose hypothesis, a renderer predicts depth measurements and a physics simulator predicts contact outcomes under guarded probing actions; both are scored against real observations to update the belief. The resulting multimodal belief also enables information-gain-based probing for active disambiguation. Across simulated geometries and real-robot experiments, BayesContact improves pose observability and insertion success over vision-only inference by 30%

Aditya Kamireddypalli, Matias Mattamala, Joao Moura +3

School of Informatics, University of Edinburgh, Scotland. · Department of Mechanical and Mechatronics Engineering, University of Waterloo, Canada.

Category-Level Object Pose Estimation Tactile Pac-Bayesian Theory Hypothesis

Jul 17, 2026cs.CL

BayesPO: Bayesian Prompt Optimization via Parallel-Tempered Gradient-Guided Discrete MCMC

Prompt optimization adapts large language models (LLMs) without updating model parameters, but many automatic prompt optimizers remain heuristic search procedures over candidate instructions. This paper studies prompt optimization as Bayesian posterior sampling over discrete prompt tokens. We define a posterior distribution by combining a task likelihood term, which rewards prompts that explain input-output examples, with a language-model prior, which favors fluent instructions. This converts prompt optimization into an energy-based posterior sampling problem, for which gradients can be used to guide discrete Markov chain Monte Carlo (MCMC) proposals over vocabulary tokens. We refer to our framework as BayesPO, short for Bayesian Prompt Optimization. In this paper, BayesPO is instantiated with Markov chain Monte Carlo: it uses a Metropolis-Hastings corrected Gibbs-with-Langevin (GwL) proposal and integrates parallel tempering for global exploration of rugged LLM-induced energy landscapes. The concrete sampler further adapts the GwL sampler to the practical constraints of non-weight-tied LLM embeddings. Experiments with Qwen2.5 models show that the sampler discovers semantically meaningful prompts on diagnostic tasks, that parallel tempering helps escape a local optimum in a poetry completion task, and that post-optimizing APE prompts on 24 instruction-induction subtasks improves average accuracy from 60.04% to 63.23%. The study also reveals two main limitations: energy minimization may overfit small optimization sets, and the current sampler remains computationally expensive. These findings position Bayesian prompt sampling as a principled post-optimization tool and point to a promising direction for probabilistic prompt optimization.

Junjie Zhou, Zhijian Ou

Speech Processing and Machine Intelligence (SPMI) Lab Tsinghua University, China

Prompt Optimization Posterior Sampling Pac-Bayesian Theory Markov Chain Monte Carlo Free-Energy Landscape Poetry

Jul 7, 2026quant-ph

Entanglement as a Structural Complexity Axis: A PAC-Bayesian View of Generalization in Quantum Policies and Value Functions

Parameterized quantum circuits (PQCs) are increasingly used as policies and value functions in quantum reinforcement learning, yet it remains unclear when and why quantum policies generalize. We give a PAC-Bayesian account in which generalization is governed not by the raw number of circuit parameters, but by the effective dimension of the Fisher geometry induced by the circuit. This quantity is inflated by entanglement, making entangling connectivity an independent axis of complexity.In controlled experiments that fix the number of trainable rotations and vary only entanglement, we find that circuits with larger Fisher effective dimension exhibit larger train-test gaps, while parameter count is a weak predictor. The resulting bound acts primarily as a ranking certificate: it correctly orders circuits with identical parameter count, which parameter-counting bounds cannot do. We validate this mechanism across supervised classification, quantum contextual bandits, and value-function generalization, where entangled circuits consistently generalize worse than non-entangled circuits of equal parameter count, with gaps shrinking as sample size increases.Our strongest evidence comes from low-variance decision models, including single-observable classifiers, value heads, and one-step policies. In end-to-end multi-step policy learning, entanglement effects remain statistically significant but high return variance leaves the full ordering only partially resolved. Partial-correlation analysis shows that Fisher effective dimension screens off entangling pattern, and controls for training accuracy, readout, and optimizer rule out major optimization confounders. The effect also persists on an IBM Heron quantum processor under real noise. Overall, our results reframe quantum policy design around an entanglement--generalization trade-off rather than expressivity alone.

Jian Xu, Delu Zeng, John Paisley +1

1RIKEN iTHEMS · 2RIKEN AIP · 3South China University of Technology +1

Entanglement Pac-Bayesian Theory Structural Complexity Latent Confounders

Jun 29, 2026cs.LG

Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework

We present a complete formal proof that transformer architectures, when their internal update mechanisms satisfy a Bayes joint-distribution condition, implement exact Bayesian posterior inference. Working within the measure-theoretic kernel framework, we define a hierarchy of abstractions -- from the core Bayesian transformer, through semantic transformers with explicit update kernels, to full transformer blocks with QKV/attention/residual/MLP pipelines, and finally multilayer stacks -- and prove at each level that the Bayes joint semantics implies the update kernel equals the posterior almost everywhere. For the block-level architecture, we derive the explicit Bayes formula through Radon-Nikodym differentiation and prove its normalization. We additionally prove that the softmax attention mechanism induces a valid probability distribution over keys, establishing the bridge between the abstract kernel framework and concrete attention implementations. The framework makes no architectural assumptions beyond the Markov kernel structure and exposes explicit conditions under which a transformer block is provably Bayesian. In essence, when this joint distribution condition is satisfied, the forward computation of a Transformer is formally equivalent to a rigorous Bayesian posterior update.

Haobo Yang

Department of Computer Science and Engineering, SUSTech University

Transformer Architectures Bayesian Posterior Pac-Bayesian Theory Probability Measures

Jun 29, 2026cs.AI

BayesEvolve: Explicit Belief States for Autonomous Scientific Discovery

Autonomous scientific discovery systems increasingly use large language models (LLMs) to propose new hypotheses, but many such systems condition primarily on experimental memory: archives of high-scoring candidates or heuristic summaries of recent trials. We argue that discovery agents should instead maintain explicit, uncertainty-aware beliefs about hypothesis quality. We introduce BayesEvolve, a belief-guided discovery framework that converts experimental evidence into a predictive belief state and uses this belief to guide future experimentation. As a controlled testbed for belief-guided discovery, we evaluate BayesEvolve on shifted BBOB-style black-box optimization tasks, leaving program and laboratory discovery domains to future work. BayesEvolve improves sample efficiency over memory- and archive-guided LLM baselines under a fixed evaluation budget. We further show that the belief state is predictive on held-out candidate pools, that controlled decision-rule ablations favor belief-guided selection with an annealed uncertainty bonus, and that BayesEvolve exhibits productive late-stage concentration rather than unfocused exploration.

Xuening Wu, Shan Yu, Qianya Xu +1

Pfizer, Shanghai, China · Independent Researcher, Hangzhou, China · University of California San Diego, La Jolla, CA, USA +1

Stochastic Exploration Discovery Pac-Bayesian Theory

Jun 19, 2026cs.RO

BayesFP: Posterior Estimation for Flow-Based Policies via Feynman-Kac Sampling

Robots must generate trajectories that remain faithful to learned expert behavior while satisfying safety constraints and task-specific objectives specified only at inference time. We formulate constrained trajectory generation for pretrained diffusion and flow-matching policies as Bayesian posterior sampling, with the learned demonstration distribution as a prior and an inference-time, cost-derived likelihood tilting it toward feasible, optimal trajectories. To sample from this posterior without any retraining of the base policy, we leverage the Feynman--Kac corrector framework, originally formulated for diffusion models, and extend it to deterministic flow-matching policies. The result is a unified, inference-time, retraining-free sampler for diffusion and flow policies. We validate the approach on pretrained Diffusion Policy, GR00T-N1.6, and

π_{0.5}

checkpoints across simulated and real-world manipulation tasks, including planning around non-convex obstacles introduced at inference time, and show improvements over the base

π_{0.5}

on zero-shot tasks.

Sreevardhan Sirigiri, Weiming Zhi, Fabio Ramos

School of Computer Science, The University of Sydney, Australia · Australian Centre For Robotics, The University of Sydney, Australia · College of Connected Computing, Vanderbilt University, TN, USA. +1

Flow Policies Diffusion Policies Posterior Sampling Posterior Trajectory Generation Pac-Bayesian Theory

Jun 11, 2026cs.LG

Loss-Shift Transfer via Bayes Quotients

Transfer learning is usually studied as a consequence of distribution shift. This paper identifies an orthogonal failure mode in which the data distribution is fixed and the loss changes. This setting is called \emph{loss shift}. A loss determines which information in

X

is Bayes-relevant, and two losses may therefore require different representations even under the same joint law

P(X,Y)

. The idea is formalized using Bayes quotients, which allow losses to be ordered by refinement. In the Bayes-quotient formulation, strict refinement gives an immediate qualitative obstruction. A source-minimal representation for a coarser loss is insufficient for a strictly finer target loss. For finite-output log loss, this obstruction becomes an exact quantitative identity. The excess risk is the conditional information about

Y

discarded by the representation. Experiments in controlled, learned, synthetic-image, and real-image settings show the predicted effect, i.e., classification-equivalent representations can have different optimal log-loss performance under a fixed data distribution.

Vasileios Sevetlidis

Athena Research Center, Kimmeria Campus, Xanthi, Greece · Democritus University of Thrace, Vas. Sofias Campus, Xanthi, Greece · International Hellenic University, Serres, Greece

Loss Function Distribution Shifts Pac-Bayesian Theory Transfer Learning

Jun 2, 2026cs.LG

Bayes-Sufficient Representations in Supervised Learning

Representation learning is often described as preserving the information in an input that is relevant for prediction. This work asks what relevance means for a fixed supervised decision problem. A representation is defined to be Bayes-sufficient for a joint distribution and loss if some prediction head can use it to implement a Bayes-optimal action rule. This makes the target information loss-dependent. In the almost-surely unique Bayes-action case, the relevant object is a Bayes quotient, which identifies inputs that require the same Bayes-optimal action. A representation is sufficient when it refines this quotient, and Bayes-minimal when it is informationally equivalent to it. The framework connects naturally to property elicitation: zero-one loss requires the Bayes class, squared loss the conditional mean, Brier loss the conditional probability in binary prediction, and log loss or strictly proper scoring rules the predictive distribution. Controlled finite experiments, learned neural bottleneck experiments, and a real-data iNaturalist taxonomic refinement experiment illustrate the distinction between sufficiency, minimality, and retained non-required information. For a fixed supervised problem, the distribution and the loss determine the Bayes action, the Bayes action determines the quotient, and the quotient determines the minimal information required for Bayes-optimal prediction.

Vasileios Sevetlidis

Athena Research Center, Kimmeria Campus, Xanthi, Greece · Democritus University of Thrace, Vas. Sofias Campus, Xanthi, Greece · International Hellenic University, Serres, Greece

Pac-Bayesian Theory Bayesian Representation Learning Semi-Supervised Learning Loss Function Relevance

May 25, 2026cs.LG

A PAC-Bayesian View of Generalisation for Physics-Informed Machine Learning

Physics-informed machine learning (PIML) integrates mechanistic knowledge, typically in the form of partial differential equations (PDE), into data-driven models. Despite strong empirical performance, its statistical generalisation properties remain poorly understood, particularly in the regression setting with unbounded losses. Existing analyses rely on approximation or stability arguments and do not fully capture how physical structure influences generalisation from finite data. In this work, we develop a PAC-Bayesian framework for PIML that provides high-probability generalisation guarantees in the presence of unbounded losses. We adopt a multi-task perspective that jointly treats data fidelity, PDE residuals, initial and boundary conditions, avoiding the looseness induced by standard union-bound approaches. Our analysis leverages the structure of physics-informed objectives to derive novel bounds where the complexity scales with input-gradient norms of the losses, revealing a direct link between physical regularity and generalisation. We instantiate this framework under Sobolev and Poincaré-type assumptions, yielding two classes of bounds that trade off statistical complexity and smoothness in different regimes. Building on these results, we propose a self-bounding-aware learning algorithm that directly optimises tractable surrogates of the derived bounds, along with a practical procedure to estimate the associated constants in realistic settings. Empirical evaluations on standard PDE benchmarks demonstrate that our bounds are non-vacuous, significantly tighter than union-bound baselines, and can be effectively minimised during training. Overall, our results provide a principled statistical foundation for the generalisation of physics-informed models.

Thien V. Nguyen, Amaury Habrard, Benjamin Guedj

Université Jean Monnet Saint-Étienne, CNRS, Institut d’Optique Graduate School, Laboratoire Hubert Curien UMR 5516, F-42023, Saint-Étienne, France · Université Jean Monnet Saint-Étienne, CNRS, Institut d’Optique Graduate School, Laboratoire Hubert Curien UMR 5516, Inria, F-42023, Saint-Étienne, France · Institut Universitaire de France +1

Physics-Informed Learning Pac-Bayesian Theory Partial Differential Equations Generalisation

May 21, 2026cs.NE

Guiding Multi-Objective Genetic Programming with Description Length Improves Symbolic Regression Solutions

Symbolic regression with genetic programming (GPSR) may suffer from overfitting and structural bloat, especially when noise is present. In this paper we evaluate description length (DL) and fractional Bayes factor (FBF) criteria as principled, data-efficient alternatives to heuristics for selecting compact expressions that generalise well. We implement DL using a Fisher-information-based parameter encoding and compare it to AIC and BIC across multiple datasets, including noisy synthetic benchmarks and real-world regression problems. We study three search/selection strategies: (i) multi-objective search for accuracy and program length followed by DL/FBF selection; (ii) multi-objective search using DL directly as an objective; and (iii) single-objective optimisation with DL/FBF as the fitness. Across datasets we find that DL/FBF post-selection improves test performance compared to AIC/BIC baseline and that BIC in combination with the same function complexity penalty from DL/FBF produces similar results. In contrast, using DL/FBF directly as a fitness function in single-objective GPSR frequently induces premature convergence to overly simple models. We conclude with practical guidance for using DL/FBF as robust model-selection tools in genetic programming workflows.

Gabriel Kronberger, Fabricio Olivetti de Franca, Deaglan J. Bartlett +2

Heuristic and Evolutionary Algorithms Laboratory University of Applied Sciences Upper Austria Softwarepark 11, 4232 Hagenberg, Austria · Center of Mathematics, Computing and Cognition Federal University of ABC Av. dos Estados 5001, Santo Andre, 09280-560, SP, Brazil · Astrophysics, University of Oxford Denys Wilkinson Building, Keble Road, Oxford, OX1 3RH, UK +1

Symbolic Regression Genetic Algorithms Model Selection Gaussian Process Pac-Bayesian Theory Rtk-Gnss

May 20, 2026cs.LG

MMD-Balls as Credal Sets: A PAC-Bayesian Framework for Epistemic Uncertainty in Test-Time Adaptation

Test-time adaptation (TTA) methods improve model performance under distribution shift but lack formal guarantees connecting shift magnitude to prediction reliability. We develop a PAC-Bayesian framework yielding generalization bounds explicitly parameterized by the maximum mean discrepancy (MMD) between source and target distributions. Our principal contribution is interpreting MMD-balls around the source distribution as credal sets in Walley's imprecise probability theory, yielding natural epistemic uncertainty quantification. We establish: (i) a PAC-Bayesian bound with an MMD-dependent shift penalty under an RKHS-Lipschitz loss assumption; (ii) a finite-sample version via MMD concentration; (iii) a uniform worst-case risk bound over all distributions in the credal set, with a lower-upper risk decomposition; and (iv) geodesic preservation bounds explaining why kernel-guided adaptation protects local feature geometry. The credal set interpretation separates epistemic from aleatoric uncertainty and provides a principled decision criterion for when adaptation is warranted.

Ahanaf Hasan Ariq

Ideal School and College

Pac-Bayesian Theory Empirical Risk Minimization Generalization Bounds Dynamic Epistemic Logic Integrity

May 20, 2026cs.LG

A Sharper Picture of Generalization in Transformers

We study transformers' generalization behavior on boolean domains from the perspective of the Fourier spectra of their target functions. In contrast to prior work (Edelman et al., 2022; Trauger & Tosh, 2024), which derived generalization bounds from Rademacher complexity, we investigate the feasibility of obtaining generalization bounds via PAC-Bayes theory. We show that sparse spectra concentrated on low-degree components enable low-sharpness constructions with good generalization properties. Our idea is to show the existence of flat minima implementing any boolean function of sparsity no greater than the context length, and then apply a PAC-Bayes bound to an idealized low-sharpness learner, resulting in a non-vacuous generalization bound. We use this to give a formal account of why chain-of-thought improves generalization for high-degree target functions, and show that the complexity parameters in our bound can be efficiently estimated via property testing. We evaluate predictions empirically and conduct a mechanistic interpretability study to support the realism of our theoretical construction in real transformers.

Paul Lintilhac, Sair Shaikh

Thayer School of Engineering Dartmouth College · Dartmouth College

Generalization Bounds Sharpness-Aware Minimization Transformer Architectures Learnability Pac-Bayesian Theory

May 17, 2026cs.LG

Anytime PAC-Bayes for Constrained Density-Ratio Networks under Covariate Shift

A unified framework for learning under covariate shift is presented, in which a constrained density-ratio network approximates the Radon-Nikodym derivative

r^\star = dP/dQ

and feeds an anytime PAC-Bayes generalization certificate. A change-of-measure identity decomposes the gap between target risk and importance-weighted source risk into a ratio-bias term governed by

\|r_θ- r^\star\|_{L^2(Q)}

and a generalization-gap term governed by the variability of the weighted loss. Normalization and moment-matching identities are enforced as hard integral constraints through an augmented-Lagrangian scheme, with a second-moment penalty controlling the effective sample size. PAC-Bayes is instantiated on the weighted risk in a fixed-time regime that yields Bernoulli-KL bounds, identifies the network-weighted Gibbs posterior as the unique KL-regularized minimizer, and quantifies stability under

L^2(Q)

perturbations of the learned ratio, and is then strengthened by geometric peeling to an anytime certificate uniform in

t \geq t_{\min}

. A pre-registered two-campaign protocol combining a patch test against analytic ground truth with a real-data deployment validates the framework: the network produces calibrated ratios, reduces target

0/1

loss against unweighted ERM and classical direct ratio-estimation baselines, and attains the anytime certificate. A single fixed-time coverage failure is recorded, with per-split coverage aligning one-to-one with the magnitude of the label shift, confirming that the covariate-only assumption is operationally tight rather than a defect of the certificate.

Paulo Akira F. Enabe, Rodrigo Provasi

Escola Politénica University of São Paulo Department of Structural and Geotechnical Engineering

Covariate Shift Pac-Bayesian Theory Importance Weighting Empirical Risk Minimization Gibbs

May 13, 2026stat.ML

A Survey on Data-Dependent Worst-Case Generalization Bounds

Deep neural networks generalize well despite being heavily overparameterized, in apparent contradiction with classical learning theory based on uniform convergence over fixed hypothesis spaces. Uniform bounds over the entire parameter space are vacuous in this regime, and recent work has shown that non-vacuous guarantees can be recovered by restricting attention to the part of parameter space that the algorithm actually visits. This survey paper organizes this line of work around three steps: extending PAC-Bayesian theory to random, data-dependent hypothesis sets (arXiv:2404.17442); refining the complexity term with geometric and topological descriptors of the optimization trajectory, including fractal dimensions, alpha-weighted lifetime sums, and positive magnitude (arXiv:2006.09313, arXiv:2302.02766, arXiv:2407.08723); and replacing the resulting information-theoretic terms by stability assumptions (arXiv:2507.06775). We unify these contributions around a single template inequality and a head-to-head comparison of the resulting bounds.

Hubert Leroux, Jean Marcus, Julien Roger

´Ecole polytechnique Palaiseau, France

Generalization Bounds Overparameterization Upper Bounds Pac-Bayesian Theory Hypothesis Class Survey

May 11, 2026cs.SE

Instruction Adherence in Coding Agent Configuration Files: A Factorial Study of Four File-Structure Variables

Frontier coding agents read configuration files (CLAUDE

.

md, AGENTS

.

md, Cursor Rules) at session start and are expected to follow the conventions inside them. Practitioners assume that structural choices (file size, instruction position, file architecture, contradictions in adjacent files) measurably affect adherence. We report a systematic factorial study of these choices using four manipulated variables, measuring compliance with a trivial target annotation across 1,650 Claude Code CLI sessions (16,050 function-level observations) on two TypeScript codebases, three frontier models (primarily Sonnet 4.6, with Opus 4.6 as a CLI-matched cross-model check and Opus 4.7 reported descriptively under a CLI-version confound), and five coding tasks. We use mixed-effects models with a Bayesian companion. None of the four structural variables or three two-way interactions produces a detectable contrast after multiple-testing correction. Size and conflict nulls are supported by affirmative-null Bayes factors (BF10 between 0.05 and 0.10); position and architecture nulls are failures to reject without Bayes-factor support. The largest effect we measured is within-session: each additional function the agent generates is associated with approximately 5.6% lower odds of compliance per step (OR = 0.944) within the session-length range we tested, though the relationship is non-monotonic rather than a constant per-step effect. This reproduces on a second TypeScript codebase and on Opus 4.6 at matched configuration; it was identified during analysis rather than pre-specified. Within the conditions tested, file-structure variables did not produce detectable contrasts; compliance varies systematically between coding tasks and across each session's sequence of generated functions.

Damon McMillan

HxAI Australia

Coding Agents Compliance Frontier Models Factor Instruction Pac-Bayesian Theory

May 8, 2026cs.AI

PLACO: A Multi-Stage Framework for Cost-Effective Performance in Human-AI Teams

Human-AI teams play a pivotal role in improving overall system performance when neither the human nor the model can achieve such performance on their own. With the advent of powerful and accessible Generative AI models, several mundane tasks have morphed into Human-AI team tasks. From writing essays to developing advanced algorithms, humans have found that using AI assistance has led to an accelerated work pace like never before. In classification tasks, where the final output is a single hard label, it is crucial to address the combination of human and model output. Prior work elegantly solves this problem using Bayes rule, using the assumption that human and model output are conditionally independent given the ground truth. Specifically, it discusses a combination method to combine a single deterministic labeler (the human) and a probabilistic labeler (the classifier model) using the model's instance-level and the human's class-level calibrated probabilities.

Pranavkumar Mallela, Vinay Kumar, Shashi Shekhar Jha +1

Indian Institute of Technology Ropar

Human-Ai Collaboration Teaming Label Propagation Pac-Bayesian Theory

May 7, 2026cs.CV

TinyBayes: Closed-Form Bayesian Inference via Jacobi Prior for Real-Time Image Classification on Edge Devices

Cocoa (Theobroma cacao) is a critical cash crop for millions of smallholder farmers in West Africa, where Cocoa Swollen Shoot Virus Disease (CSSVD) and anthracnose cause devastating yield losses. Automated disease detection from leaf images is essential for early intervention, yet deploying such systems in resource-constrained settings demands models that are small, fast, and require no internet connectivity. Existing edge-deployable plant disease systems rely on end-to-end deep learning without uncertainty quantification, while Bayesian methods for edge devices focus on hardware-level inference architectures rather than agricultural applications. We bridge this gap with TinyBayes, the first framework to combine a closed-form Bayesian classifier with a mobile-grade computer vision pipeline for crop disease detection. Our pipeline uses YOLOv8-Nano (5.9 MB) for lesion localisation, MobileNetV3-Small (3.5 MB) for feature extraction, and the Jacobi prior; a Bayesian method that provides a closed form non-iterative estimators via projection, for the classification. The Jacobi-DMR (Distributed Multinomial Regression) classifier adds only 13.5 KB to the pipeline, bringing the total model size within 9.5 MB, while achieving 78.7% accuracy on the Amini Cocoa Contamination Challenge dataset and enabling end-to-end CPU inference under 150 ms per image. We benchmark against seven classifiers including Random Forest, SVM, Ridge, Lasso, Elastic Net, XGBoost, and Jacobi-GP, and demonstrate that the Jacobi-DMR offers the best trade-off between accuracy, model size, and inference speed for edge deployment. We have proved the asymptotic equivalence and consistency, asymptotic normality and the bias correction of Jacobi-DMR. All data and codes are available here: https://github.com/shouvik-sardar/TinyBayes

Shouvik Sardar, Sourish Das

Data Science Group Chennai Mathematical Institute

Leaf Disease Classification Edge Devices Pac-Bayesian Theory Deep Learning Prior Knowledge Integration Jacobian

May 5, 2026cs.LG

Realizable Bayes-Consistency for General Metric Losses

We study strong universal Bayes-consistency in the realizable setting for learning with general metric losses, extending classical characterizations beyond

0

-

1

classification (Bousquet et al., 2020; Hanneke et al., 2021) and real-valued regression (Attias et al., 2024). Given an instance space

(X,ρ)

, a label space

(Y,\ell)

with possibly unbounded loss, and a hypothesis class

H \subseteq Y^{X}

, we resolve the realizable case of an open problem presented in Tsir Cohen and Kontorovich (2022). Specifically, we find the necessary and sufficient conditions on the hypothesis class

H

under which there exists a distribution-free learning rule whose risk converges almost surely to the best-in-class risk (which is zero) for every realizable data-generating distribution. Our main contribution is this sharp characterization in terms of a combinatorial obstruction: Similarly to Attias et al. (2024), we introduce the notion of an infinite non-decreasing

(γ_k)

-Littlestone tree, where

γ_k \to \infty

. This extends the Littlestone tree structure used in Bousquet et al. (2020) to the metric loss setting.

Dan Tsir Cohen, Steve Hanneke, Aryeh Kontorovich

1Ben-Gurion University of the Negev · 2Purdue University, USA.

Hypothesis Class Pac-Bayesian Theory Metric Spaces Loss Function

Apr 28, 2026cs.LG

Teacher Forcing as Generalized Bayes: Optimization Geometry Mismatch in Switching Surrogates for Chaotic Dynamics

Identity teacher forcing (ITF) enables stable training of deterministic recurrent surrogates for chaotic dynamical systems and has been highly effective for dynamical systems reconstruction (DSR) with recurrent neural networks (RNNs), including interpretable almost-linear RNNs (AL-RNNs). However, as an intervention-based prediction loss (and thus a generalized Bayes update), teacher forcing need not match the free-running model's marginal likelihood geometry. We compare the objective-induced curvatures of ITF and marginal likelihood in a probabilistic switching augmentation of AL-RNNs, estimating ambiguity-aware observed information via Louis' identity. In the switching setting studied here, conditioning on a single forced regime path (as ITF does) inflates curvature, while marginal likelihood curvature is reduced by a missing-information correction when multiple switching explanations remain plausible. In Lorenz-63 experiments, windowed evidence fine-tuning improves held-out evidence but can degrade dynamical quantities of interest (QoIs) relative to ITF-pretrained models.

Andre Herz, Daniel Durstewitz, Georgia Koppe

Interdisciplinary Center for Scientific Computing, Heidelberg University, Germany · Dept. of Theoretical Neuroscience, Central Institute of Mental Health (CIMH), Mannheim, Germany · Hector Institute for AI in Psychiatry and Dept. of Psychiatry and Psychotherapy, CIMH, Mannheim, Germany +1

Chaos Pac-Bayesian Theory Teacher Surrogates

Apr 27, 2026cs.LG

Null Measurability at the Symmetrization Interface in VC Learning

Recent work revisiting measurability in the fundamental theorem of statistical learning imposes Borel measurability of ghost-gap suprema. We show that, at the one-sided ghost-gap interface actually used by the standard symmetrization proof, this requirement is stronger than necessary. For any Borel-parameterized concept class on a Polish domain, the bad event "there exists a hypothesis whose ghost empirical error exceeds its training empirical error by at least ε/2" is analytic. By Choquet capacitability, it is therefore measurable in the completion of every finite Borel measure. We then construct a concept class whose bad event is null-measurable but not Borel, giving a strict separation from the Borel supremum condition. Finally, we prove closure under patching, fixed and countable interpolation, and fiber-product amalgamation, showing that the weaker regularity level is stable under natural concept-class constructors. In the realizable setting, where targets belong to the class and are measurable, these results weaken the measurability hypothesis needed by the symmetrization route from finite VC dimension to PAC learnability. The main results and the descriptive-set-theoretic infrastructure used by them are formalized in Lean 4.

Dhruv Gupta

AI and Robotics Technology Park, Indian Institute of Science, Bengaluru

Probability Measures Learnability Pac-Bayesian Theory Measures Symmetry Voice Conversion

Apr 19, 2026stat.ML

PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory

We derive explicit non-asymptotic PAC-Bayes generalization bounds for Gibbs posteriors, that is, data-dependent distributions over model parameters obtained by exponentially tilting a prior with the empirical risk. Unlike classical worst-case complexity bounds based on uniform laws of large numbers, which require explicit control of the model space in terms of metric entropy (integrals), our analysis yields posterior-averaged risk bounds that can be applied to overparameterized models and adapt to the data structure and the intrinsic model complexity. The bound involves a marginal-type integral over the parameter space, which we analyze using tools from singular learning theory to obtain explicit and practically meaningful characterizations of the posterior risk. Applications to low-rank matrix completion and ReLU neural network regression and classification show that the resulting bounds are analytically tractable and substantially tighter than classical complexity-based bounds. Our results highlight the potential of PAC-Bayes analysis for precise finite-sample generalization guarantees in modern overparameterized and singular models.

Chenyang Wang, Yun Yang

Department of Statistics, University of Illinois Urbana-Champaign · Department of Mathematics, University of Maryland, College Park

Pac-Bayesian Theory Generalization Bounds Posterior Overparameterization Singular Learning Theory Structural Complexity

Apr 17, 2026cs.LG

When Do Early-Exit Networks Generalize? A PAC-Bayesian Theory of Adaptive Depth

Early-exit neural networks enable adaptive computation by allowing confident predictions to exit at intermediate layers, achieving 2-8

\times

inference speedup. Despite widespread deployment, their generalization properties lack theoretical understanding -- a gap explicitly identified in recent surveys. This paper establishes a unified PAC-Bayesian framework for adaptive-depth networks. (1) Novel Entropy-Based Bounds: We prove the first generalization bounds depending on exit-depth entropy

H(D)

and expected depth

\mathbb{E}[D]

rather than maximum depth

K

, with sample complexity

\mathcal{O}((\mathbb{E}[D] \cdot d + H(D))/ε^2)

. (2) Explicit Constructive Constants: Our analysis yields the leading coefficient

\sqrt{2\ln 2} \approx 1.177

with complete derivation. (3) Provable Early-Exit Advantages: We establish sufficient conditions under which adaptive-depth networks strictly outperform fixed-depth counterparts. (4) Extension to Approximate Label Independence: We relax the label-independence assumption to

ε

-approximate policies, broadening applicability to learned routing. (5) Comprehensive Validation: Experiments across 6 architectures on 7 benchmarks demonstrate tightness ratios of 1.52-3.87

\times

(all

p < 0.001

) versus

>

100

\times

for classical bounds. Bound-guided threshold selection matches validation-tuned performance within 0.1-0.3%.

Dongxin Guo, Jikun Wu, Siu Ming Yiu

The University of Hong Kong, Hong Kong, China · Brain Investing Limited, Hong Kong, China

Generalization Bounds Pac-Bayesian Theory

Feb 26, 2026cs.LG

Bound to Disagree: Generalization Bounds via Certifiable Surrogates

Generalization bounds for deep learning models are typically vacuous, not computable or restricted to specific model classes. In this paper, we tackle these issues by providing new disagreement-based certificates for the gap between the true risk of any two predictors. We then bound the true risk of the predictor of interest via a surrogate model that enjoys tight generalization guarantees, and by evaluating our disagreement bound on an unlabeled dataset.We empirically demonstrate the tightness of the obtained certificates and showcase the versatility of the approach by training surrogate models leveraging three different frameworks: sample compression, model compression and PAC-Bayes theory. Importantly, such guarantees are achieved without modifying the target model, nor adapting the training procedure to the generalization framework.

Mathieu Bazinet, Valentina Zantedeschi, Pascal Germain

1Département d’informatique et génie logiciel, Université Laval, Québec, Qc, Canada · 2ServiceNow Research, Montreal, Qc, Canada

Generalization Bounds Surrogate Models Pac-Bayesian Theory Deep Learning Surrogates Disagreement

Nov 20, 2025cs.AI

MedBayes-Lite: A Clinical Uncertainty Governance Layer for Risk-Aware Medical Decision Support

Clinical language models often assign high confidence to incorrect predictions, particularly in high-severity and out-of-distribution cases. We present MedBayes-Lite, a retraining-free uncertainty governance layer for transformer-based clinical predictors. It combines Monte Carlo dropout, predictive calibration, and confidence-guided abstention to defer low-confidence predictions for human review, adding no trainable parameters. Evaluated on MedMCQA and MedQA-USMLE, MedBayes-Lite reduces expected calibration error by 0.23 to 0.33 and drives harmful overconfident errors (confident, incorrect, high-severity predictions) toward zero. Under domain shift from MedMCQA to MedQA-USMLE, it reduces confident high-severity errors from about 21% to near zero while roughly halving calibration drift. We also introduce the Clinical Uncertainty Score (CUS), which strongly correlates with harmful overconfidence (r approximately 0.88). Although the framework does not improve risk-coverage ranking, and temperature scaling or deep ensembles may provide advantages in calibration cost or risk ranking, MedBayes-Lite offers a practical calibration-and-abstention layer that reduces confident high-severity errors in clinical question-answering benchmarks.

Elias Hossain, Md Mehedi Hasan Nipu, Maleeha Sheikh +5

College of Engineering and Computer Science, University of Central Florida, Orlando, FL 32816, USA · Department of Computer Science and Engineering, North South University, Dhaka 1229, Bangladesh · Department of Electrical and Computer Engineering, Purdue University Fort Wayne, Fort Wayne, IN 46805, USA +4

Clinical Decision Support Large Language Model Uncertainty Uncertainty Pac-Bayesian Theory

Oct 10, 2024cs.LG

How Learning Dynamics Drive Adversarially Robust Generalization?

Despite being widely adopted as a canonical framework for learning robust models, adversarial training suffers from robust overfitting. Existing empirical and theoretical explorations fail to provide a satisfactory mechanistic interpretation of the phenomenon. By modeling adversarial training with momentum SGD as a discrete-time dynamical system, we propose a PAC-Bayesian analytical framework that proves time-resolved robust generalization bounds. Specifically, our framework tracks the closed-form evolution of the posterior mean and covariance under both stationary and non-stationary transient regimes, connecting the model's robust generalization performance to learning rate, local loss geometry, and mini-batch stochastic gradients. By estimating the key quantities associated with the bound, we illustrate the underlying mechanism of robust overfitting. Our framework also shows how adversarial weight perturbation reduces robust generalization gaps by suppressing dominant loss-curvature modes, while suggesting that excessive penalization can be sub-optimal for optimization.

Yuelin Xu, Xiao Zhang

CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Adversarial Training Generalization Bounds Learnability Pac-Bayesian Theory Momentum Stochastic Gradient Descent