cs.LGJul 28, 2025

Diagonally-Weighted Generalized Method of Moments Estimation for Gaussian Mixture Modeling

Authors: Liu Zhang, Oscar Mickelin, Sheng Xu, Amit Singer

Organizations: Program in Applied and Computational Mathematics, Princeton University, Princeton, NJ 08540 USA · Department of Mathematics and Program in Applied and Computational Mathematics, Princeton University, Princeton, NJ 08540 USA

Abstract

With direct computation of moments through order LL in dd dimensions, standard method of moments (MM) incurs Ω(dL)Ω(d^L) computation and storage costs, and standard generalized method of moments (GMM) additionally requires estimating and inverting an often ill-conditioned covariance matrix of size Ω(dL)Ω(d^L)-by-Ω(dL)Ω(d^L) to obtain the optimal weighting matrix. Pereira, Kileel, and Kolda [arXiv:2202.06930 (2022)] avoid the tensor computation cost of MM through implicit moment computation. Despite this progress, incorporating statistically principled weighting while preserving computational efficiency remains an important challenge, particularly at low signal-to-noise ratio (SNR), where sampling variances can differ substantially across moment orders. We introduce diagonally-weighted GMM (DGMM), which assigns one adaptive weight to each moment order through a structured approximation of the optimal weighting matrix while avoiding the covariance inversion, thereby balancing statistical efficiency, computational complexity, and numerical stability. For general parametric models, we prove that DGMM is consistent, asymptotically normal, and near-efficient under explicit structural conditions. For heteroscedastic low-rank Gaussian mixtures, we design a computationally efficient and numerically stable algorithm that extends the implicit moment computation to the DGMM objective, gradients, and weights and incorporates Nyström approximations of the kernel sums. In low-SNR experiments with d=100d=100 and L=4L=4, where standard MM and GMM are infeasible, controlled weighting comparisons show that DGMM achieves consistently small parameter errors across tested initializations, including the challenging starts where alternative weightings struggle, with shorter median runtimes and fewer iterations. The code and data will be available upon publication at https://github.com/liu-lzhang/dgmm.

Figures & tables

Explore similar work

May 26, 2026

Robust Moment-Based Estimation via Spectral Gradient Reweighting

Moment-based estimation is a theoretically attractive approach to parametric inference, especially when likelihood-based estimation is unavailable, misspecified, or computationally inconvenient. However, the moment equations involve sample averages, which makes moment-based estimation sensitive to outliers. We propose the SGR-GMM algorithm, a robust generalized method of moments (GMM) procedure that uses a spectral gradient reweighting (SGR) primitive to soft-reweight the per-observation gradients during the moment-matching optimization. Our analysis has three layers. First, for a fixed center, the SGR primitive is formulated as an entropy-regularized spectral game between a sample-weight player and a density-matrix player, which is analyzed using classical multiplicative-weights and matrix-multiplicative-weights regret bounds. Second, we establish explicit convergence radius and finite termination bound for the fixed-center updates in the SGR primitive. Third, we prove a local finite-sample parameter estimation error bound with explicit dependence on the contamination fraction, inlier gradient stability, local GMM identification strength, and optimization accuracy. We further specialize the SGR-GMM algorithm to obtain a robust diagonally-weighted GMM (DGMM) estimator for estimating heteroscedastic low-rank Gaussian mixtures observed under additive Gaussian noise and strong contamination. In the numerical experiments, the SGR primitive produces nearly-oracle gradient estimation and the robust DGMM specialization substantially improves over non-robust moment baselines. The code and data are available at https://github.com/liu-lzhang/sgr-gmm.
Jan 21, 2025

Sublinear Variational Optimization of Gaussian Mixture Models with Millions to Billions of Parameters

Gaussian Mixture Models (GMMs) range among the most frequently used models in machine learning. However, training large, general GMMs becomes computationally prohibitive for data sets that have many data points NN of high-dimensionality DD. For GMMs with arbitrary covariances, we here derive a highly efficient variational approximation, which is then integrated with mixtures of factor analyzers (MFAs). For GMMs with CC components, our proposed algorithm substantially reduces runtime complexity from O(NCD2)\mathcal{O}(NCD^2) per iteration to a complexity scaling linearly with DD and sublinearly with NCNC. In numerical experiments, we first validate that the complexity reduction results in a sublinear scaling for the entire GMM optimization process. Second, we show on large-scale benchmarks that the sublinear algorithm results in speed-ups of an order-of-magnitude compared to the state-of-the-art. Third, as a proof of concept, we finally train GMMs with over 10 billion parameters on about 100 million images, observing training times of less than nine hours on a single state-of-the-art CPU. Finally, and fourth, we demonstrate the effectiveness of large-scale GMMs on the task of zero-shot image denoising, where sublinear training results in state-of-the-art denoising times while competitive denoising performance is maintained.
Mar 20, 2026

Model Selection and Parameter Estimation for Multidimensional Gaussian Mixture Models with a Common Covariance Matrix

We study model-order selection and component-mean estimation for multidimensional Gaussian mixture models with a known common covariance matrix. Using empirical characteristic-function measurements, we construct Fourier covariance matrices whose population counterparts have rank equal to the number of mixture components. We establish a minimax lower bound showing that distinguishing a separated kk-component mixture from the class of (k−1)(k-1)-component mixtures requires Ω(Δ−(4k−4))Ω(Δ^{-(4k-4)}) samples. We then develop an oracle spectral-thresholding estimator with a sufficient sample size of order Δ−(8k−8)Δ^{-(8k-8)} for fixed kk, together with a practical singular-value-ratio estimator. Given the model order, we estimate the component means by score-initialized gradient descent on a MUSIC-type projection objective. Under an explicit sample-size condition, a qualifying sample initialization lies in a certified attraction region with high probability, after which the iterates converge linearly. For fixed positive component separation, the resulting mean estimates achieve the parametric rate Op(n−1/2)\mathcal{O}_p(n^{-1/2}). Numerical experiments demonstrate competitive accuracy and lower computational cost than expectation-maximization across a range of multidimensional settings.