cs.LGApr 5, 2026

Multirate Stein Variational Gradient Descent for Efficient Bayesian Sampling

Authors: Arash Sarshar

Organizations: Department of Computer Engineering and Computer Science, California State University, Long Beach, 90840, CA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Many particle-based Bayesian inference methods use a single global step size for all parts of the update. In Stein variational gradient descent (SVGD), however, each update combines two qualitatively different effects: attraction toward high-posterior regions and repulsion that preserves particle diversity. These effects can evolve at different rates, especially in high-dimensional, anisotropic, or hierarchical posteriors, so one step size can be unstable in some regions and inefficient in others. We derive a multirate version of SVGD that updates these components on different time scales. The framework yields practical algorithms, including a symmetric split method, a fixed multirate method (MR-SVGD), and an adaptive multirate method (Adapt-MR-SVGD) with local error control. We evaluate the methods in a broad and rigorous benchmark suite covering six problem families: a 50D Gaussian target, multiple 2D synthetic targets, UCI Bayesian logistic regression, multimodal Gaussian mixtures, Bayesian neural networks, and large-scale hierarchical logistic regression. Evaluation includes posterior-matching metrics, predictive performance, calibration quality, mixing, and explicit computational cost accounting. Across these six benchmark families, multirate SVGD variants improve robustness and quality-cost tradeoffs relative to vanilla SVGD. The strongest gains appear on stiff hierarchical, strongly anisotropic, and multimodal targets, where adaptive multirate SVGD is usually the strongest variant and fixed multirate SVGD provides a simpler robust alternative at lower cost.

Figures & tables

Figure 1: 50D Gaussian final-checkpoint summary across methods. The four panels report ∥μ^−μ∥2\|\hat{\mu}-\mu\|_{2} , ∥Σ^−Σ∥F\|\hat{\Sigma}-\Sigma\|_{F} , KSD, and ESS, respectively. Markers show mean values across seeds and error bars indicate one standard deviation. Lower is better for the first three metrics, whereas higher is better for ESS. This figure highlights robustness differences under strong anisotropy: Adapt-MR-SVGD is the only particle variant that keeps both moment errors and KSD under control.
Figure 2: 50D Gaussian: final mean ±\pm std Pareto plot in moment-error space with marker size encoding ESS (left) and wall time (right).
Figure 3: 2D target visualization panels produced with Adapt-MR-SVGD under visualization-only settings. Each subfigure shows target-density contours with initial particles (left) and short-run final particles (right).
MethodBest KSD ↓\downarrowmean logp@best ↑\uparrowgrad@best ↓\downarrowwall@best (s) ↓\downarrow
Banana
Strang-SVGD0.114-0.4261001.35
MR-SVGD0.114-0.392801.13
Adapt-MR-SVGD0.114-0.3931600.80
SVGD0.114-0.391800.32
Funnel
Table 1: 2D particle-method summary over five seeds. For each target and method, we report medians over seeds at the best finite-KSD checkpoint. Methods are ordered by median KSD within each target block. Banana shows little separation, funnel remains nuanced, and Adapt-MR-SVGD is strongest on ring, squiggly, and two moons.
Figure 4: Mixture2D (mix8) fixed-budget final-checkpoint summary across methods. The four panels report mode coverage, mode entropy, mode imbalance, and KSD. Higher is better for coverage and entropy, while lower is better for mode imbalance and KSD.
Figure 5: UCI logistic regression summary across datasets. Each subpanel reports test accuracy, NLL, ECE, and ESS at the best finite-NLL checkpoint selected by the predictive stopping rule. Remaining datasets are shown in the continued panel.
Figure 6: UCI logistic regression summary across datasets (continued).
MethodBest NLL ↓\downarrowAcc@best NLL ↑\uparrowWall@best (s) ↓\downarrow
Breast cancer
Adapt-MR-SVGD0.0280.9873.7
SVGD0.0910.9791.4
Strang-SVGD0.0960.9794.0
MR-SVGD0.0960.9793.4
Ionosphere
Table 2: UCI logistic-regression particle-method summary. For each dataset and method, we average over the finite best-NLL checkpoints selected per seed and report NLL, the corresponding accuracy, and wall time. ECE is summarized in the per-dataset panels of fig. 5 .
Figure 7: BNN predictive summary across datasets. Each subpanel reports test accuracy, NLL, ECE, and ESS at the checkpoint selected by validation NLL.
MethodFinite seeds ↑\uparrowBest NLL ↓\downarrowAcc@best NLL ↑\uparrowWall@best (s) ↓\downarrow
Adapt-MR-SVGD5/50.6130.658123.3
MR-SVGD5/50.6230.65692.5
SGHMC2/50.8600.56077.9
SVGD3/50.8850.577154.2
Strang-SVGD3/51.2630.579163.6
SGLD4/51.8230.576101.6
Table 3: HLR long-tail summary over five seeds. For each seed and method, we select the best finite-NLL checkpoint and report method-wise means of that NLL, the accuracy at the selected checkpoint, and the corresponding wall time. “Finite seeds” counts runs that reached at least one finite-NLL checkpoint before stopping. This is the main hierarchical stress test: Adapt-MR-SVGD attains the lowest mean best-NLL while remaining finite on all seeds, and MR-SVGD is the closest lower-cost baseline.
MethodFinite seeds ↑\uparrowBest NLL ↓\downarrowAcc@best NLL ↑\uparrowWall@best (s) ↓\downarrow
Adapt-MR-SVGD5/50.6810.60194.6
SVGD1/50.7000.54854.9
MR-SVGD5/50.7060.59744.2
Strang-SVGD1/50.7080.584241.7
SGLD5/50.7430.52866.0
SGHMC5/50.7910.55479.7
Table 4: HLR uniform-group summary over five seeds. Entries are defined as in table 3 , but with balanced group assignments as a sensitivity check.

Explore similar work

CardsList

  1. Improved Finite-Particle Convergence Rates for Stein Variational Gradient Descent

    Sep 13, 2024Sayan Banerjee, Krishnakumar Balasubramanian, Promit GhosalKernel MethodsVariational Inference

    Department of Statistics and Operations Research, University of North Carolina, Chapel Hill. · Department of Statistics, University of California, Davis. · Department of Statistics, University of Chicago.

  2. Variational Inference via Entropic Transport Descent

    Jun 24, 2026Vincent Pacelli, Akash Ratheesh, Evangelos TheodorouBayesian InferenceVariational Inference

    School of Aerospace Engineering Georgia Institute of Technology Atlanta, GA 30332

  3. Robust Stochastic Gradient Posterior Sampling with Lattice Based Discretisation

    Feb 17, 2026Zier Mensch, Lars Holdijk, Samuel Duffield +4Posterior SamplingMarkov Chain Monte Carlo

    Institute of Physics, University of Amsterdam, Netherlands · Department of Physics, National Taiwan University, Taiwan · Department of Computer Science, University of Oxford, United Kingdom +4