Expectation-Maximization

Also known as EM

Overview

Expectation-Maximization (EM) estimates models with hidden variables by alternating an E-step, which infers latent-variable distributions, and an M-step, which updates model parameters. Its variants include generalized and gradient EM, stochastic and online EM, and variational EM, which trades exact inference for scalability. These methods are used in mixture models, hidden Markov models, incomplete-data imputation, and imaging, among other settings. A central challenge is that EM can depend on initialization and converge to local optima; stronger guarantees often require conditions such as component separation, while faster or approximate updates raise questions about accuracy and stability.

Latest papers

31 papers4 papers in the last four weeks, down 20% on the four weeks before. 0.0% of all new papers.

Jul 14, 2019

Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts

Mixtures of experts (MoE) are conditional mixture models in which both the mixing proportions and the component densities depend on the predictors, and are widely used for regression, classification and model-based clustering of heterogeneous data. Fitting MoE by maximum likelihood becomes unstable, and sometimes infeasible, when the predictors are numerous or correlated. We propose a regularized maximum likelihood framework for simultaneous parameter estimation and feature selection in MoE whose experts belong to the generalized linear model family, covering Gaussian, Poisson and multinomial responses within a single formulation. Sparsity is induced in both the gating network and the experts through ℓ1\ell_1 penalties, and the penalized log-likelihood is maximized by a proximal Newton-EM algorithm whose M-step reduces to weighted Lasso problems with closed-form coordinate-ascent updates. Unlike existing penalized MoE procedures, the algorithm requires neither a local quadratic approximation of the penalty nor any matrix inversion, it returns exactly sparse estimates without thresholding, and a proximal Newton-type variant guarantees a monotone increase of the penalized objective at every iteration. On simulated data and five real data sets, the method recovers the actual sparsity support and delivers prediction and clustering accuracy that is competitive with, and often better than, state-of-the-art regularized MoE. The source codes of our developed algorithms and their documentation are publicly available on Github at https://github.com/nv-thin/GLM-RMoE.