Stability of Sequential and Parallel Coordinate Ascent Variational Inference

Debdeep Pati

stat.MLMar 21, 2026

Stability of Sequential and Parallel Coordinate Ascent Variational Inference

Authors: Debdeep Pati

Organizations: Department of Statistics, University of Wisconsin-Madison, Madison, WI 53706

Abstract

We highlight a striking difference in behavior between two widely used variants of coordinate ascent variational inference: the sequential and parallel algorithms. While such differences were known in the numerical analysis literature in simpler settings, they remain largely unexplored in the optimization-focused literature on variational inference in more complex models. Focusing on the moderately high-dimensional linear regression problem, we show that the sequential algorithm, although typically slower, enjoys convergence guarantees under more relaxed conditions than the parallel variant, which is often employed to facilitate block-wise updates and improve computational efficiency.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of sequential (left) and parallel (right) versions for $p=2$ .

Figure 2: Sequential CAVI for

(n,p,s)=(200,50,25)

. Left: variational mean

\mu_{j}

(red) versus true coefficients

\beta_{j}

(black). Right: ELBO as a function of iteration.

Figure 3: Distribution of

\log\rho(J)

for sequential and parallel CAVI. Left: varying

p

with

s=p

and fixed

n=100

. Right: varying

s

with fixed

(n=200,p=50)

Explore similar work

Wasserstein Contraction of Coordinate Ascent Variational Inference
May 28, 2026Rocco Caprio, Adrien Corenflos, Sam PowerBayesian Inference Variational Inference
University of Warwick and Bocconi University. · University of Warwick. · University of Bristol.
Stochastic Gradient Variational Inference with Price's Gradient Estimator from Bures-Wasserstein to Parameter Space
Feb 21, 2026Kyurae Kim, Qiang Fu, Yi-An Ma +2Variational Inference
University of Pennsylvania, Philadelphia, U.S. · Yale University, New Haven, U.S. · University of California San Diego, San Diego, U.S. +1
Adaptive Delayed-Update Cyclic Algorithm for Variational Inequalities
Mar 31, 2026Yi Wei, Xufeng Cai, Jelena Diakonikolas
Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI

May 28, 2026J/K move · Enter open · S save

Wasserstein Contraction of Coordinate Ascent Variational Inference

Rocco Caprio, Adrien Corenflos, Sam Power

University of Warwick and Bocconi University. · University of Warwick. · University of Bristol.

Bayesian Inference Variational Inference Wasserstein Distance

Open paper AskPDF arXiv

We study the non-asymptotic contraction in Wasserstein distance of the sequential, parallel, and random-scan coordinate ascent variational inference algorithms. This is shown to hold under a functional smoothness condition of the optimality maps and a transportation-information inequality at their fixed points. Our results are sharp and general, and as opposed to those based on global strong log-concavity assumptions, they allow for local convergence on smooth, non-smooth, and discrete manifolds, including within the context of data augmentation. We consider many applications in statistical physics and Bayesian statistics. These include pairwise Markov Random field models such as Ising and Curie-Weiss, unbalanced Bayesian Gaussian Mixture Models, high-dimensional Bayesian Probit Regression, and high-dimensional Logistic Regression with Pólya--Gamma random variables (i.e. Jaakkola-Jordan's algorithm). In many of these models, these represent the first available convergence results of their kind.

Abstract

Explore similar work

Wasserstein Contraction of Coordinate Ascent Variational Inference

Stochastic Gradient Variational Inference with Price's Gradient Estimator from Bures-Wasserstein to Parameter Space

Adaptive Delayed-Update Cyclic Algorithm for Variational Inequalities