stat.MLMay 24, 2026

Estimating Mixture Distributions via Stochastic Mirror Descent

Authors: Mohammadreza Ahmadypour, Tara Javidi, Farinaz Koushanfar

Organizations: Department of Electrical and Computer Engineering University of California San Diego, San Diego, CA, USA

Abstract

We revisit the classical problem of estimating an unknown distribution from its samples by fitting a mixture model that minimizes cross-entropy loss. Framing the task as a stochastic convex optimization problem over the space of MM-component mixture distributions, we propose a family of estimators derived from the stochastic mirror descent (SMD) algorithm. This optimization-based approach provides a principled and flexible framework that generalizes traditional estimators and proposes a variety of novel estimators through the choice of Bregman divergences. A key advantage of our method is that it scales efficiently with the number of candidate components fif_i; that is, one can employ a large set of basis distributions in the mixture model without incurring significant computational overhead. This enables richer approximations and improved estimation accuracy. Moreover, in the case of categorical distribution (discrete outcomes) our estimators do not require a strict lower bound, in other words our framework does not require the precise knowledge of the support of the distribution. We demonstrate that, under mild conditions, the proposed φ\varphi-SMD estimators achieve near-optimal convergence rates in both Kullback-Leibler (KL) divergence and ℓ2\ell_2-norm and offer practical benefits when computation is expensive. Our numerical analysis highlights improved performance guaranties over classical estimators, particularly in terms of sample efficiency and scalability.

Explore similar work

CardsList
Sep 16, 2024stat.ML

The EM-algorithm and the Method of Moments in Softmax Mixture Models

Softmax Mixture Models (SMMs) are discrete KK-component mixture models for the probabilities of selecting one of pp candidate feature vectors X1,…,Xp∈RLX_1,\ldots,X_p\in\mathbb{R}^L in heterogeneous populations and are widely used in econometrics and scientific applications. Related softmax mixture mechanisms also appear in modern LLM architectures. We provide a theoretical and methodological study of SMMs, focusing on the Expectation-Maximization (EM) algorithm and the Method of Moments (MoM). We show that EM recovers the mixture atoms at the parametric rate, up to logarithmic factors, after O(log⁡N)\mathcal{O}(\log N) iterations, provided atom separation is at least of order log⁡K\log K. This improves on separation conditions in existing analyses of EM for high-dimensional Gaussian mixtures. We also develop MoM procedures for parameter and subspace estimation. Although MoM parameter estimates converge more slowly than EM and can deteriorate with KK, they provide provable warm starts for EM and are useful for small KK. For general KK, we estimate the atom subspace via MoM and recommend running EM from multiple random initializations within this subspace. Finally, as p→∞p\to\infty, we show that SMMs approximate mixtures of exponential tilts of the feature distribution, yielding asymptotic identifiability.
Xin Bing, Florentina Bunea, Jonathan Niles-Weed +1
Mixture ModelsExpectation-MaximizationGumbel-Softmax RelaxationDiffie-Hellman
AskPDF
Jun 9, 2026cs.DS

Density estimation for Hellinger via minimum-distance estimators: mixtures of Gaussians, log-concave, and more

We study the task of density estimation, where we hope to accurately estimate a probability density from nn samples. A textbook method for density estimation in total variation distance is the minimum-distance estimator approach, where we conclude both the algorithm and the analysis merely from bounding the VC dimension of a particular concept class (the so-called Yatracos class). While this technique has originally yielded sharp guarantees primarily for total variation distance, in this work we extend the minimum-distance estimator approach for learning within Hellinger distance. Our main observation is that we may produce an analogous recipe for Hellinger (where we only require bounding the VC dimension of a related concept class) by drawing connections to recent results yielding reverse data processing inequalities. This recipe is flexible enough to accommodate fast algorithms originally designed for total variation distance; by modifying the approach of Acharya et al. (2017) we conclude the first near-linear time algorithm for learning classes including univariate mixtures of log-concave densities and mixtures of Gaussians (with arbitrary variances), with near-optimal sample complexity.
Spencer Compton, Jerry Li
Randomized Global-Local Density EstimationOptimal Sample ComplexityTotal VariationEstimatorsMixtures3D Gaussian Splatting
AskPDF
May 20, 2026cs.LG

Symbolic Density Estimation for Discrete Distributions

Discrete probability laws underpin statistical modeling, yet the catalog of interpretable distributions has expanded only gradually through centuries of case-by-case mathematical derivations. We introduce symbolic density estimation (SDE), an unsupervised framework that automatically recovers closed-form probability mass functions by composing elementary analytic operations within a structured search space. Our method integrates domain-specific structural priors with evolutionary search and a validity-aware inference stage, and it extends to richer distribution families such as zero inflation and finite mixtures. To support systematic evaluation and future research, we contribute a benchmark dataset spanning a broad collection of commonly used discrete distributions. The proposed algorithm recovers all benchmark families with accurate parameter estimates. A real data application shows that it identifies concise and interpretable mixture models that improve goodness-of-fit over standard models.
Ziwen Liu, Meng Li
Randomized Global-Local Density EstimationConditional DistributionDistributional Learning
AskPDF