cs.LGJul 28, 2025

Diagonally-Weighted Generalized Method of Moments Estimation for Gaussian Mixture Modeling

Authors: Liu Zhang, Oscar Mickelin, Sheng Xu, Amit Singer

Organizations: Program in Applied and Computational Mathematics, Princeton University, Princeton, NJ 08540 USA · Department of Mathematics and Program in Applied and Computational Mathematics, Princeton University, Princeton, NJ 08540 USA

Abstract

With direct computation of moments through order LL in dd dimensions, standard method of moments (MM) incurs Ω(dL)Ω(d^L) computation and storage costs, and standard generalized method of moments (GMM) additionally requires estimating and inverting an often ill-conditioned covariance matrix of size Ω(dL)Ω(d^L)-by-Ω(dL)Ω(d^L) to obtain the optimal weighting matrix. Pereira, Kileel, and Kolda [arXiv:2202.06930 (2022)] avoid the tensor computation cost of MM through implicit moment computation. Despite this progress, incorporating statistically principled weighting while preserving computational efficiency remains an important challenge, particularly at low signal-to-noise ratio (SNR), where sampling variances can differ substantially across moment orders. We introduce diagonally-weighted GMM (DGMM), which assigns one adaptive weight to each moment order through a structured approximation of the optimal weighting matrix while avoiding the covariance inversion, thereby balancing statistical efficiency, computational complexity, and numerical stability. For general parametric models, we prove that DGMM is consistent, asymptotically normal, and near-efficient under explicit structural conditions. For heteroscedastic low-rank Gaussian mixtures, we design a computationally efficient and numerically stable algorithm that extends the implicit moment computation to the DGMM objective, gradients, and weights and incorporates Nyström approximations of the kernel sums. In low-SNR experiments with d=100d=100 and L=4L=4, where standard MM and GMM are infeasible, controlled weighting comparisons show that DGMM achieves consistently small parameter errors across tested initializations, including the challenging starts where alternative weightings struggle, with shorter median runtimes and fewer iterations. The code and data will be available upon publication at https://github.com/liu-lzhang/dgmm.

Figures & tables

Explore similar work

CardsList
  1. Robust Moment-Based Estimation via Spectral Gradient Reweighting

    May 26, 2026Liu Zhang, Amit SingerRobust StatisticsRobust Estimation

  2. Sublinear Variational Optimization of Gaussian Mixture Models with Millions to Billions of Parameters

    Jan 21, 2025Sebastian Salwig, Till Kahlke, Florian Hirschberger +2Gaussian Mixture ModelsVariational Inference

  3. Model Selection and Parameter Estimation for Multidimensional Gaussian Mixture Models with a Common Covariance Matrix

    Mar 20, 2026Xinyu Liu, Hai ZhangGaussian Mixture ModelsSpectral Methods