Momentum SVGD-EM for Accelerated Maximum Marginal Likelihood Estimation

Adam Rozzio; Rafael Athanasiades; O. Deniz Akyildiz

stat.MLMar 9, 2026

Momentum SVGD-EM for Accelerated Maximum Marginal Likelihood Estimation

Authors: Adam Rozzio, Rafael Athanasiades, O. Deniz Akyildiz

Organizations: Department of Mathematics, Ecole Normale Supérieure Paris-Saclay, France · Department of Mathematics, Imperial College London, UK

Abstract

Maximum marginal likelihood estimation (MMLE) can be formulated as the optimization of a free energy functional. From this viewpoint, the Expectation-Maximisation (EM) algorithm admits a natural interpretation as a coordinate descent method over the joint space of model parameters and probability measures. Recently, a significant body of work has adopted this perspective, leading to interacting particle algorithms for MMLE. In this paper, we propose an accelerated version of one such procedure, based on Stein variational gradient descent (SVGD), by introducing Nesterov acceleration in both the parameter updates and in the space of probability measures. The resulting method, termed Momentum SVGD-EM, consistently accelerates convergence in terms of required iterations across various tasks of increasing difficulty, demonstrating effectiveness in both low- and high-dimensional settings.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of M-SVGD-EM with the SVGD-EM . Part (a) shows the performance on parameter estimation on a single iteration on the ToyHM(10, 12) with different acceleration parameters $\alpha_{\theta}=\alpha_{X}=(0.3,0.5,0.9)$ .(b) displays MSE with respect to the empirical minimizer for each algorithm. (c) shows the average number of iterations until convergence, defined as being within a threshold of 0.05 from the empirical minimizer. The results are averaged over 20 independent trials.

Panel of Figure 2 — Figure 2: (a) Parameter estimation convergence on BLR varying acceleration. (b) Posterior distributions across test features, showing acceleration improves convergence. (c) Test error vs. acceleration for different $t$ values, showing error reduction with increased acceleration.

Panel of Figure 3 — Figure 3: Different Initilisations of $\alpha$ and $\beta$ for accelerations $(0,0.3,0.9)$ with $N=10$ for $T=500$ .

Panel of Figure 4 — Figure 4: Test Error w.r.t iterations $t$ for the BNN on MNIST dataset

Panel of Figure 5 — Figure 5: Log predictive probability distribution vs iterations $t$ for the BNN on MNIST dataset

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Comparison between M-SVGD-EM and SVGD-EM . Part (a) shows the performance on parameter estimation on a single run on the ToyHM(10, 12) with different acceleration parameters $\alpha_{\theta}=\alpha_{X}=(0.3,0.5,0.9)$ . Part (b) displays MSE relative to the empirical minimizer for each algorithm. (c) shows the average number of iterations until convergence, defined as being within a threshold of 0.05 from the empirical minimizer. The results are averaged over 20 independent trials.

Panel of Figure 7 — Figure 7: MSE vs Learning Rate

Panel of Figure 8 — Figure 8: (a) Convergence of parameter estimates on BLR under varying acceleration parameters. (b) Posterior distributions across 4 test features. (c) Test error as a function of the acceleration parameter for different values of $t$ , showing consistent error reduction with increased acceleration.

Panel of Figure 9 — Figure 9: Test Error (Accuracy) for Bayesian Logistic Regression with different numbers of particles.

Panel of Figure 10 — Figure 10: Performance metrics for the BNN on MNIST dataset: (a) Test Error across iterations $t$ , and (b) Log Predictive Probability Density (LPPD).

Figure 11: Test Error using

N={5,10,20}

with initial

\alpha=\beta=0.0

Figure 12: Log Predictive Probability Density using

N={5,10,20}

with initial

\alpha=\beta=0.0

Figure 13: Test Error using

N={5,10,20}

with initial

\alpha=\beta=2.0

Figure 14: Log Predictive Probability Density using

N={5,10,20}

with initial

\alpha=\beta=2.0

Panel of Figure 15 — Figure 15: MSE w.r.t Learning Rate over 10 independent trials.

Panel of Figure 16 — Figure 16: Parameter estimation of a single run varying dimensions $D=5,20,50$ .

Panel of Figure 17 — Figure 17: Log-Log Parameter estimation of a single run varying dimensions $D=5,20,50$ .

Panel of Figure 18 — Figure 18: MSE vs iterations calculated w.r.t the empirical minimizer $D=5,20,50$ .

Panel of Figure 19 — Figure 19: Average number of iterations $\pm$ se for 20 independent trials $D=5,20,50$ .

Panel of Figure 20 — Figure 20: Test Error for Bayesian Logistic Regression with different numbers of particles.

Figure 21: Heatmap Toy Hierarchical Model $\gamma$ vs $\mu$ ,with the darker color giving the least iterations needed until convergence

Figure 22: Heatmap Wisconsin Bayesian Logistic Regression Model $\gamma$ vs $\mu$ ,with the darker color giving the least iterations needed until convergence

Notebook	Time (s)
Toy Hierarchical Model	$497.09$
Bayesian Logistic Regression	$423.35$
Bayesian Neural Network	$578.71$

Appendix

Table 1: Average Runtime for each model’s notebook

Maximum Marginal Likelihood Estimation

See all 11 papers

A Gradient Flow Perspective on Minimum MMD Estimation
Jul 4, 2026Sophia Seulkee Kang, Louis Sharrock, Xiaoyuan Cheng +2Parameter Estimation Maximum Mean Discrepancy
Independent Researcher · University College London
The EM-algorithm and the Method of Moments in Softmax Mixture Models
Sep 16, 2024Xin Bing, Florentina Bunea, Jonathan Niles-Weed +1Expectation-Maximization Discrete Choice Modeling
Department of Statistical Sciences, University of Toronto · Department of Statistics and Data Science, Cornell University · Courant Institute of Mathematical Sciences and Center for Data Science, New York University +1
Multirate Stein Variational Gradient Descent for Efficient Bayesian Sampling
Apr 5, 2026Arash SarsharStein Variational Gradient Descent Bayesian Inference
Department of Computer Engineering and Computer Science, California State University, Long Beach, 90840, CA, USA

Jul 4, 2026J/K move · Enter open · S save

A Gradient Flow Perspective on Minimum MMD Estimation

Sophia Seulkee Kang, Louis Sharrock, Xiaoyuan Cheng, François-Xavier Briol+1

Independent Researcher · University College London

Parameter Estimation Maximum Mean Discrepancy Gradient Descent Stochastic Optimization Convergence

Open paper AskPDF arXiv

Minimum maximum mean discrepancy (MMD) estimation has emerged as a robust and likelihood-free alternative to maximum likelihood estimation for parameter estimation. Yet, despite its practical success, the associated optimization problem remains poorly understood, with theoretical guarantees for existing algorithms hinging on convexity assumptions that rarely hold in practice. We address this gap by proposing a preconditioned gradient descent (PGD) scheme, establishing its asymptotic global convergence under explicit gradient-dominance and projection-residual conditions. Our approach is inspired by recent progress on MMD gradient flows, a nonparametric descent scheme on the space of probability measures. We provide extensive empirical evidence that our PGD scheme outperforms standard gradient descent across a range of challenging parameter estimation and composite hypothesis testing problems.

Abstract

Maximum Marginal Likelihood Estimation

Explore similar work

A Gradient Flow Perspective on Minimum MMD Estimation

The EM-algorithm and the Method of Moments in Softmax Mixture Models

Multirate Stein Variational Gradient Descent for Efficient Bayesian Sampling