Thermodynamic Response Functions in Singular Bayesian Models

Sean Plummer

stat.MLMar 5, 2026

Thermodynamic Response Functions in Singular Bayesian Models

Authors: Sean Plummer

Abstract

Singular statistical models-including mixtures, matrix factorization, and neural networks-violate regular asymptotics due to parameter non-identifiability and degenerate Fisher geometry. Although singular learning theory characterizes marginal likelihood behavior through invariants such as the real log canonical threshold and singular fluctuation, these quantities remain difficult to interpret operationally. At the same time, widely used criteria such as WAIC and WBIC appear disconnected from underlying singular geometry. We show that posterior tempering induces a one-parameter deformation of the posterior distribution whose associated observables generate a hierarchy of thermodynamic response functions. A universal covariance identity links derivatives of tempered expectations to posterior fluctuations, placing WAIC, WBIC, and singular fluctuation within a unified response framework. Within this framework, classical quantities from singular learning theory acquire natural thermodynamic interpretations: RLCT governs the leading free-energy slope, singular fluctuation corresponds to curvature of the tempered free energy, and WAIC measures predictive fluctuation. We formalize an observable algebra that quotients out non-identifiable directions, allowing structurally meaningful order parameters to be constructed in singular models. Across canonical singular examples-including symmetric Gaussian mixtures, reduced-rank regression, and overparameterized neural networks-we empirically demonstrate phase-transition-like behavior under tempering. Order parameters collapse, susceptibilities peak, and complexity measures align with structural reorganization in posterior geometry. Our results suggest that thermodynamic response theory provides a natural organizing framework for interpreting complexity, predictive variability, and structural reorganization in singular Bayesian learning.

Figures & tables

Panel of Figure 1 — Figure 1: Order parameters and susceptibilities arise from the covariance identity. The response-speed bound links rates of change to fluctuation magnitudes and connects naturally to heat capacity.

Figure 2: Response hierarchy for the mixture symmetry-breaking experiment. The top panel shows the order parameter $m(\beta)=\mathbb{E}_{\beta}[|\mu|]$ , which measures the posterior preference for one component mean over the symmetric configuration. At low inverse temperature the posterior explores both symmetric modes. As $\beta$ increases the posterior concentrates on one mode, producing spontaneous symmetry breaking. The middle panel shows the susceptibility $\chi(\beta)=\beta\mathrm{Var}(|\mu|)$ , which peaks near the transition where the posterior fluctuates between symmetric configurations. The bottom panel shows the WAIC complexity $\log(1+p_{\mathrm{WAIC}}(\beta)/n)$ . Predictive variance decreases as the posterior concentrates, indicating reduced predictive uncertainty once symmetry is broken. The vertical dashed line marks the temperature where susceptibility is maximal.

Figure 3: Response hierarchy for reduced-rank regression. The order parameter $m(\beta)=\mathbb{E}_{\beta}[s_{2}(B)]$ tracks the second singular value of the regression matrix. As $\beta$ increases, posterior concentration drives the second singular value toward zero, indicating collapse to a lower-rank model. The susceptibility $\chi(\beta)=\beta\mathrm{Var}(s_{2})$ measures fluctuations in the effective rank and peaks near the temperature where rank collapse occurs. The WAIC complexity decreases as the posterior eliminates redundant directions in parameter space. The alignment between susceptibility and predictive complexity illustrates how singular structure controls predictive variability.

Figure 4: Response hierarchy for the neural network hidden-unit collapse experiment. The order parameter $m(\beta)=\mathbb{E}_{\beta}[N_{\mathrm{eff}}]$ measures the effective number of active hidden units. Although the network contains $H=10$ units, the posterior favors a smaller effective number as $\beta$ increases. Redundant hidden units become inactive due to symmetry and scaling degeneracies. The susceptibility $\chi(\beta)=\beta\mathrm{Var}(N_{\mathrm{eff}})$ peaks when multiple configurations with different numbers of active units coexist. This region corresponds to maximal posterior uncertainty over network representations. The WAIC complexity decreases as redundant units collapse, indicating that predictive uncertainty is highest when the network’s internal representation is unstable.

Singular Learning Theory

Singular Learning Theory studies how non-identifiability and geometric singularities shape learning, Bayesian inference, and generalization.

See all 16 papers

Aug 20, 2026

Exact Algebraic Computation of Learning Coefficients for Two-Dimensional Singular Models

Classical information criteria such as the Bayesian Information Criterion (BIC) rely on regularity assumptions that break down for singular models, leading to incorrect model selection in settings such as deep learning. The Widely Applicable Bayesian Information Criterion (WBIC) relies on local learning coefficients

λ

, which in the analytic case coincides with local Real Log Canonical Thresholds (RLCT) of the Kullback-Leibler divergence of the model, to capture correct marginal likelihood asymptotics. Exact computation of the learning coefficients has been limited to special cases, and only sampling-based estimation methods are generally applicable. We present the first deterministic algorithm that computes local RLCTs exactly for any two-dimensional model whose Kullback-Leibler distance is contact equivalent to a polynomial, derive a bound on its complexity, and demonstrate its effectiveness for a broad class of models, with applications including polynomial neural networks. Beyond providing ground truth to calibrate sampling-based estimators, exact computation reveals algebraic structure in learning coefficients that sampling cannot and out-speeds it in the shallow regime.

Grégoire Sergeant-Perthuis, Elias Tsigaridas, Jules Tsukahara

CQSB, Sorbonne Université · Ouragan Team, INRIA · Ouragan team, Inria Paris

Singular Learning Theory

Ask PDF

Apr 19, 2026

PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory

We derive explicit non-asymptotic PAC-Bayes generalization bounds for Gibbs posteriors, that is, data-dependent distributions over model parameters obtained by exponentially tilting a prior with the empirical risk. Unlike classical worst-case complexity bounds based on uniform laws of large numbers, which require explicit control of the model space in terms of metric entropy (integrals), our analysis yields posterior-averaged risk bounds that can be applied to overparameterized models and adapt to the data structure and the intrinsic model complexity. The bound involves a marginal-type integral over the parameter space, which we analyze using tools from singular learning theory to obtain explicit and practically meaningful characterizations of the posterior risk. Applications to low-rank matrix completion and ReLU neural network regression and classification show that the resulting bounds are analytically tractable and substantially tighter than classical complexity-based bounds. Our results highlight the potential of PAC-Bayes analysis for precise finite-sample generalization guarantees in modern overparameterized and singular models.

Chenyang Wang, Yun Yang

Department of Statistics, University of Illinois Urbana-Champaign · Department of Mathematics, University of Maryland, College Park

Neural Network Generalization PAC-Bayesian Generalization Bounds Singular Learning Theory Statistical Learning Theory PAC-Bayesian Theory

Ask PDF

May 8, 2026

Susceptibilities and Patterning: A Primer on Linear Response in Bayesian Learning

These notes introduce the theory of susceptibilities as developed in [arXiv:2504.18274, arXiv:2601.12703] for interpreting neural networks. The susceptibility of an observable

φ

to a data perturbation is defined as a derivative of a posterior expectation, which by the fluctuation--dissipation theorem equals a posterior covariance. Different choices of

φ

yield different objects: per-sample losses give the influence matrix (the Bayesian influence function of [arXiv:2509.26544]), while component-localized observables give the structural susceptibility matrix that pairs model components with data patterns. The susceptibility matrix is (up to a factor of

nβ

) the Jacobian of the map from data distributions to structural coordinates; its pseudo-inverse provides a linearized solution to the patterning problem of [arXiv:2601.13548]: finding data perturbations that produce a desired structural change. We motivate the theory from its statistical-mechanical foundations, then give a detailed exposition of susceptibilities, their empirical estimators, and their connection to the geometry of the loss landscape.

Chris Elliott, Daniel Murfet

Timaeus

Statistical Physics of Learning Bayesian Inference Neural Network Interpretability

Ask PDF

Abstract

Singular Learning Theory

Explore similar work

Exact Algebraic Computation of Learning Coefficients for Two-Dimensional Singular Models

PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory

Susceptibilities and Patterning: A Primer on Linear Response in Bayesian Learning