cs.LGApr 5, 2026

Multirate Stein Variational Gradient Descent for Efficient Bayesian Sampling

Authors: Arash Sarshar

Organizations: Department of Computer Engineering and Computer Science, California State University, Long Beach, 90840, CA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Many particle-based Bayesian inference methods use a single global step size for all parts of the update. In Stein variational gradient descent (SVGD), however, each update combines two qualitatively different effects: attraction toward high-posterior regions and repulsion that preserves particle diversity. These effects can evolve at different rates, especially in high-dimensional, anisotropic, or hierarchical posteriors, so one step size can be unstable in some regions and inefficient in others. We derive a multirate version of SVGD that updates these components on different time scales. The framework yields practical algorithms, including a symmetric split method, a fixed multirate method (MR-SVGD), and an adaptive multirate method (Adapt-MR-SVGD) with local error control. We evaluate the methods in a broad and rigorous benchmark suite covering six problem families: a 50D Gaussian target, multiple 2D synthetic targets, UCI Bayesian logistic regression, multimodal Gaussian mixtures, Bayesian neural networks, and large-scale hierarchical logistic regression. Evaluation includes posterior-matching metrics, predictive performance, calibration quality, mixing, and explicit computational cost accounting. Across these six benchmark families, multirate SVGD variants improve robustness and quality-cost tradeoffs relative to vanilla SVGD. The strongest gains appear on stiff hierarchical, strongly anisotropic, and multimodal targets, where adaptive multirate SVGD is usually the strongest variant and fixed multirate SVGD provides a simpler robust alternative at lower cost.

Figures & tables

Figure 1: 50D Gaussian final-checkpoint summary across methods. The four panels report ∥μ^−μ∥2\|\hat{\mu}-\mu\|_{2} , ∥Σ^−Σ∥F\|\hat{\Sigma}-\Sigma\|_{F} , KSD, and ESS, respectively. Markers show mean values across seeds and error bars indicate one standard deviation. Lower is better for the first three metrics, whereas higher is better for ESS. This figure highlights robustness differences under strong anisotropy: Adapt-MR-SVGD is the only particle variant that keeps both moment errors and KSD under control.
Figure 2: 50D Gaussian: final mean ±\pm std Pareto plot in moment-error space with marker size encoding ESS (left) and wall time (right).
Figure 3: 2D target visualization panels produced with Adapt-MR-SVGD under visualization-only settings. Each subfigure shows target-density contours with initial particles (left) and short-run final particles (right).
MethodBest KSD ↓\downarrowmean logp@best ↑\uparrowgrad@best ↓\downarrowwall@best (s) ↓\downarrow
Banana
Strang-SVGD0.114-0.4261001.35
MR-SVGD0.114-0.392801.13
Adapt-MR-SVGD0.114-0.3931600.80
SVGD0.114-0.391800.32
Funnel
Table 1: 2D particle-method summary over five seeds. For each target and method, we report medians over seeds at the best finite-KSD checkpoint. Methods are ordered by median KSD within each target block. Banana shows little separation, funnel remains nuanced, and Adapt-MR-SVGD is strongest on ring, squiggly, and two moons.
Figure 4: Mixture2D (mix8) fixed-budget final-checkpoint summary across methods. The four panels report mode coverage, mode entropy, mode imbalance, and KSD. Higher is better for coverage and entropy, while lower is better for mode imbalance and KSD.
Figure 5: UCI logistic regression summary across datasets. Each subpanel reports test accuracy, NLL, ECE, and ESS at the best finite-NLL checkpoint selected by the predictive stopping rule. Remaining datasets are shown in the continued panel.
Figure 6: UCI logistic regression summary across datasets (continued).
MethodBest NLL ↓\downarrowAcc@best NLL ↑\uparrowWall@best (s) ↓\downarrow
Breast cancer
Adapt-MR-SVGD0.0280.9873.7
SVGD0.0910.9791.4
Strang-SVGD0.0960.9794.0
MR-SVGD0.0960.9793.4
Ionosphere
Table 2: UCI logistic-regression particle-method summary. For each dataset and method, we average over the finite best-NLL checkpoints selected per seed and report NLL, the corresponding accuracy, and wall time. ECE is summarized in the per-dataset panels of fig. 5 .
Figure 7: BNN predictive summary across datasets. Each subpanel reports test accuracy, NLL, ECE, and ESS at the checkpoint selected by validation NLL.
MethodFinite seeds ↑\uparrowBest NLL ↓\downarrowAcc@best NLL ↑\uparrowWall@best (s) ↓\downarrow
Adapt-MR-SVGD5/50.6130.658123.3
MR-SVGD5/50.6230.65692.5
SGHMC2/50.8600.56077.9
SVGD3/50.8850.577154.2
Strang-SVGD3/51.2630.579163.6
SGLD4/51.8230.576101.6
Table 3: HLR long-tail summary over five seeds. For each seed and method, we select the best finite-NLL checkpoint and report method-wise means of that NLL, the accuracy at the selected checkpoint, and the corresponding wall time. “Finite seeds” counts runs that reached at least one finite-NLL checkpoint before stopping. This is the main hierarchical stress test: Adapt-MR-SVGD attains the lowest mean best-NLL while remaining finite on all seeds, and MR-SVGD is the closest lower-cost baseline.
MethodFinite seeds ↑\uparrowBest NLL ↓\downarrowAcc@best NLL ↑\uparrowWall@best (s) ↓\downarrow
Adapt-MR-SVGD5/50.6810.60194.6
SVGD1/50.7000.54854.9
MR-SVGD5/50.7060.59744.2
Strang-SVGD1/50.7080.584241.7
SGLD5/50.7430.52866.0
SGHMC5/50.7910.55479.7
Table 4: HLR uniform-group summary over five seeds. Entries are defined as in table 3 , but with balanced group assignments as a sensitivity check.

Explore similar work

CardsList
Sep 13, 2024

Improved Finite-Particle Convergence Rates for Stein Variational Gradient Descent

We provide finite-particle convergence rates for the Stein Variational Gradient Descent (SVGD) algorithm in the Kernelized Stein Discrepancy (KSD\mathsf{KSD}) and Wasserstein-2 metrics. Our key insight is that the time derivative of the relative entropy between the joint density of NN particle locations and the NN-fold product target measure, starting from a regular initial distribution, splits into a dominant negative part' proportional to $N$ times the expected $\mathsf{KSD}^2$ and a smaller positive part'. This observation leads to KSD\mathsf{KSD} rates of order 1/N1/\sqrt{N}, in both continuous and discrete time, providing a near optimal (in the sense of matching the corresponding i.i.d. rates) double exponential improvement over the recent result by Shi and Mackey (2024). Under mild assumptions on the kernel and potential, these bounds also grow polynomially in the dimension dd. By adding a bilinear component to the kernel, the above approach is used to further obtain Wasserstein-2 convergence in continuous time. For the case of `bilinear + Matérn' kernels, we derive Wasserstein-2 rates that exhibit a curse-of-dimensionality similar to the i.i.d. setting. We also obtain marginal convergence and long-time propagation of chaos results for the time-averaged particle laws.

Sayan Banerjee, Krishnakumar Balasubramanian, Promit Ghosal

Department of Statistics and Operations Research, University of North Carolina, Chapel Hill. · Department of Statistics, University of California, Davis. · Department of Statistics, University of Chicago.

Kernel MethodsVariational InferenceStein Variational Gradient DescentWasserstein Distance
AskPDF
Jun 24, 2026

Variational Inference via Entropic Transport Descent

Particle-based variational inference (ParVI) methods approximate an intractable target distribution by evolving an ensemble of interacting samples. Existing approaches rely predominantly on kernel-based repulsion (e.g., SVGD), which suffers from variance collapse in high dimensions and mode collapse on multimodal targets -- pathologies caused by the absence of global transport structure. We introduce entropic transport descent (ETD), a ParVI family that frames each particle update as an entropy-regularized optimal transport problem. Derived from the JKO proximal scheme by lifting to the space of couplings and relaxing via the KL chain rule, each ETD iteration reduces to a Sinkhorn computation. The resulting transport plan provides global coordination, guiding each particle to nearby high-density proposals and naturally preserving multimodal structure. ETD can operate entirely score-free, requiring only pointwise evaluations of the unnormalized target density. Experiments on variance-collapse diagnostics, Bayesian logistic regression, neural networks, and molecular Boltzmann distributions show that ETD matches or outperforms SVGD, AGF-SVGD, and SGLD, with the largest gains in high-dimensional and multimodal settings.

Vincent Pacelli, Akash Ratheesh, Evangelos Theodorou

School of Aerospace Engineering Georgia Institute of Technology Atlanta, GA 30332

Bayesian InferenceVariational InferenceOptimal Transport
AskPDF
Feb 17, 2026

Robust Stochastic Gradient Posterior Sampling with Lattice Based Discretisation

Stochastic-gradient MCMC methods enable scalable Bayesian posterior sampling but often suffer from sensitivity to minibatch size and gradient noise. To address this, we propose Stochastic Gradient Lattice Random Walk (SGLRW), an extension of the Lattice Random Walk discretization. Unlike conventional Stochastic Gradient Langevin Dynamics (SGLD), SGLRW introduces stochastic noise only through the off-diagonal elements of the update covariance; this yields greater robustness to minibatch size while retaining asymptotic correctness. Furthermore, as comparison we analyze a natural analogue of SGLD utilizing gradient clipping. Experimental validation on Bayesian regression and classification demonstrates that SGLRW remains stable in regimes where SGLD fails, including in the presence of heavy-tailed gradient noise, and matches or improves predictive performance.

Zier Mensch, Lars Holdijk, Samuel Duffield +4

Institute of Physics, University of Amsterdam, Netherlands · Department of Physics, National Taiwan University, Taiwan · Department of Computer Science, University of Oxford, United Kingdom +4

Posterior SamplingMarkov Chain Monte CarloStochastic Gradient Langevin DynamicsStochastic Gradient Descent
AskPDF