cs.AIOct 27, 2026

Rènyi-Regularized Flow-Based Variational Inference for Bayesian Calibration

Authors: Exaucè Luweh Adjim Ngarti, Élise Arnaud, Léo Nicoletti, Victor Trappler, Arthur Vidard

Organizations: AIRSEA · Mines Saint-Étienne MSE, LIMOS, FAYOL-ENSMSE, FAYOL-ENSMSE

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Variational inference with normalizing flows provides a flexible posterior approximation for Bayesian model calibration, but struggles when the posterior has several modes separated by regions of low probability mass. We study this behavior in a two-parameter twin experiment and regularize the variational objective. Under Gaussian observation errors and a Gaussian prior, the corresponding minimization objective is the expected Tikhonov calibration objective minus the differential entropy of the approximating distribution. However, the expected Tikhonov term favors concentration near one of the modes, while entropy maximization does not guarantee multimodality. We introduce a regularization term based on Rényi divergence that encourages the approximate posterior to cover the prior support. In the connected bimodal case, a flow approximation represents both modes; in the case with well-separated modes, the flow and Gaussian mixture frequently miss one mode. The introduced regularization reduces the near-collapse fraction from 50% to 25%. While the regularization improves mode coverage in this experiment, it changes the Bayesian variational objective, introduces a trade-off in approximation accuracy, and does not provide a general solution to mode collapse.

More in this topic

Variational Inference

Variational inference approximates complex Bayesian posteriors with tractable distributions that support faster inference and uncertainty estimates.

See all 150 papers

Explore similar work

CardsList
Oct 20, 2025

Mode Collapse of Mean-Field Variational Inference

Mean-field variational inference (MFVI) is a widely used method for approximating high-dimensional probability distributions by product measures. It has been empirically observed that MFVI optimizers often suffer from mode collapse. For example, when the target measure ππ is a mixture π=wP0+(1−w)P1π= w P_0 + (1 - w) P_1, the MFVI optimizer tends to place most of its mass near a single component of the mixture. This work provides the first theoretical explanation of mode collapse in MFVI. Starting from two-component mixtures, we introduce the notion of \emph{ε\varepsilon-separateness} to quantify the separation of the mixture components. We derive explicit bounds on the mass that any MFVI optimizer assigns to each component, when P0P_0 and P1P_1 are ε\varepsilon-separated for sufficiently small ε\varepsilon. We further extend this analysis to mixture target distributions with multiple components. Our results reveal that the occurrence of mode collapse crucially depends on the relative position of the components. To address this issue, we propose rotational variational inference (RoVI), which augments MFVI with a rotation matrix. The numerical studies support our theoretical findings and illustrate improved mode coverage with RoVI in Bayesian mixture posteriors, including a real-data example.

Shunan Sheng, Bohan Wu, Alberto González-Sanz

Columbia University, Department of Statistics · Department of Statistics, Columbia University

Variational InferenceMode CollapseBayesian Mixture Models
AskPDF
Oct 2, 2022

GFlowNets and variational inference

This paper builds bridges between two families of probabilistic algorithms: (hierarchical) variational inference (VI), which is typically used to model distributions over continuous spaces, and generative flow networks (GFlowNets), which have been used for distributions over discrete structures such as graphs. We demonstrate that, in certain cases, VI algorithms are equivalent to special cases of GFlowNets in the sense of equality of expected gradients of their learning objectives. We then point out the differences between the two families and show how these differences emerge experimentally. Notably, GFlowNets, which borrow ideas from reinforcement learning, are more amenable than VI to off-policy training without the cost of high gradient variance induced by importance sampling. We argue that this property of GFlowNets can provide advantages for capturing diversity in multimodal target distributions.

Esmeralda S. Whitammer, Salem Lahlou, Tristan Deleu +5

Mila, Universit´e de Montr´eal · Google Research · Mila, Universit´e de Montr´eal, CIFAR

Generative Flow NetworksRL for Generative ModelsVariational Inference
AskPDF
Oct 18, 2024

Predictive variational inference: Learn the predictively optimal posterior distribution

Vanilla variational inference finds an optimal approximation to the Bayesian posterior distribution, but even the exact Bayesian posterior is often not meaningful under model misspecification. We propose predictive variational inference (PVI): a general inference framework that seeks and samples from an optimal posterior density such that the resulting posterior predictive distribution is as close to the true data generating process as possible, while this closeness is measured by multiple scoring rules. By optimizing the objective, the predictive variational inference is generally not the same as, or even attempting to approximate, the Bayesian posterior, even asymptotically. Rather, we interpret it as implicit hierarchical expansion. Further, the learned posterior uncertainty detects heterogeneity of parameters among the population, enabling automatic model diagnosis. This framework applies to both likelihood-exact and likelihood-free models. We demonstrate its application in real data examples.

Jinlin Lai, Antonio Linero, Yuling Yao

College of Information and Computer Sciences, University of Massachusetts Amherst · Department of Statistics and Data Sciences, University of Texas, Austin

Model MisspecificationVariational Inference
AskPDF