Expectation-Maximization

Also known as EM

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 19

Oct 6, 2026stat.ML

Is $\sqrt{d}$ Separation Necessary for Gradient EM to Learn Gaussian Mixtures in High Dimensions?

Learning Gaussian mixture models (GMMs) using the Expectation-Maximization (EM) algorithm and its gradient-based variants is a fundamental problem in machine learning. It is known that randomly initialized (gradient) EM fails to learn multi-component GMMs in the exact-parameterized setting, where the number of components matches that of the ground-truth GMM. Recently, global convergence of gradient EM has been established in the over-parameterized setting, where more components are used, provided that the ground-truth components are well separated. In particular, the minimum separation between ground-truth components is required to scale as

Ω(\sqrt{d})

, where

d

is the dimension. In this paper, we show that this dimensional dependence is unavoidable in high-dimensional settings. Specifically, we consider a hybrid EM algorithm that uses standard EM updates for the mixing weights and gradient EM updates for the component means. For any

ε> 0

, we prove that when the dimension is sufficiently large, in the worst case a separation of order

Ω(d^{0.5-ε})

is insufficient to guarantee global convergence of population gradient EM in sub-exponential time under random initialization, even in the over-parameterized regime. Our result establishes an almost optimal worst-case lower bound on the ground-truth separation required for learning Gaussian mixtures via gradient EM in high dimensions.

Yiran Zhang, Mo Zhou, Weihang Xu +2

University of California, Berkeley · University of Washington · Amazon, Inc.

Gaussian Mixture Models Expectation-Maximization Statistical Learning Theory

Oct 4, 2026cs.LG

Joint Estimation of Common-Slope Decay Rates and Spatial Amplitudes Using Parameterized Nonnegative Matrix Factorization

We formulate joint estimation of common-slope decay rates and amplitudes from room impulse responses (RIRs) as parameterized nonnegative matrix factorization with the Itakura--Saito divergence as the loss function (IS-NMF). Estimation at each short-time Fourier transform frequency bin produces detailed reverberation time (RT) curves directly from RIR powers with no backward integration needed. Standard space-alternating generalized expectation-maximization (SAGE) algorithm yields closed-form amplitude updates and a convex subproblem for each decay rate update. To accelerate estimation, we introduce contribution-weighted SAGE, which emphasizes observations where each component contributes strongly to the modeled power. Experiments with synthetic data show accurate recovery of well-separated decays and faster loss reduction than standard SAGE. Application to measured coupled-room RIRs yields frequency-dependent RT curves and reveals complementary space-time contributions of the shared decay components.

Jeremy B. Bai, Filip Elvander, Sebastian J. Schlecht

Multimedia Comms. & Signal Proc., Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany · Dept. of Information and Communications Engineering, Aalto University, Finland

Non-Negative Matrix Factorization Expectation-Maximization Room Acoustics

Sep 22, 2026math.ST

Statistical Rates for Entropic Optimal Transport in the Discrete to SubGaussian Regime

We study statistical rates in entropic optimal transport in the semi-discrete regime where one measure has finite support and the other is subGaussian. Our main result establishes parametric convergence rates for the empirical dual potentials to their population counterparts, with no dimension dependence in the leading term. Our result relies on tailored strong concavity analysis of the semi-dual objective, coupled with specialized bounds for the semi-discrete potentials. As a consequence, we obtain fast rates for downstream quantities derived from the optimal coupling. Chiefly, the empirical barycentric projection achieves a squared-error rate

n^{-1}

, matching the fully compact case and improving over the less favorable

n^{-1/2}

rate known for fully subGaussian settings. Altogether, these results may indicate a lower complexity adaptation phenomenon whereby the statistical complexity of the barycentric projection is governed by the discrete measure. As an application, we analyze Sinkhorn-EM, an EM-type algorithm in which the E-step is replaced by an entropic optimal transport problem. In a well-specified and balanced two-component Gaussian mixture model, we prove

\sqrt{n}

-consistency of the empirical iterates to their population counterparts for any fixed number of iterations, matching classical EM rates up to a

\sqrt{\log n}

factor. Simulations support the theory.

Tomas Gonzalez, Gonzalo Mena

Department of Machine Learning, Carnegie Mellon University · Department of Statistics & Data Science, Carnegie Mellon University

Expectation-Maximization Optimal Transport

Sep 14, 2026cs.LG

Impute-EM: Native Mixed-State Diffusion Models for Heterogeneous Data Imputation

Missing values are ubiquitous in heterogeneous data mining, where numerical, categorical, and binary variables often coexist. Many imputation methods, especially diffusion-based ones, treat discrete variables through continuous surrogates such as one-hot relaxations rather than modeling them natively. This creates a mismatch between the model state space and the mixed discrete and continuous structure of the data. We propose Impute-EM, an Expectation Maximization style framework that alternates between imputing missing entries with the current model and refitting a diffusion backbone on completed data. We instantiate Impute-EM with native mixed-state diffusion backbones for heterogeneous data, combining Gaussian and masked categorical components without one-hot relaxations. In exact settings, we characterize the update and show that the observed mask-indexed marginals match the targets at the limit, while making explicit that the full data distribution is generally non-identifiable from incomplete observations alone. Empirically, Impute-EM delivers the best distributional fidelity on mixed-type tabular imputation, on which downstream modeling relies, with text imputation serving as a controlled validation of the native discrete backbone.

Sergei Kholkin, Kirill Sokolov, Dmitry Baranchuk +2

Applied AI Institute · Applied AI Institute, Moscow State University · Yandex Research +1

Incomplete Data Imputation Expectation-Maximization Diffusion Models

Aug 30, 2026cs.LG

Structure Aware Neural Architecture Search for Mixture of Experts

Neural Architecture Search (NAS) has so far rarely been applied to Mixture-of-Experts (MoE) models, and existing MoE designs leave the alignment between experts and the structure of the data to emerge on its own. We propose an architecture search framework that makes this alignment an explicit search variable: the assignment of data clusters to experts is optimised jointly with the per-expert architectures. We cast the joint problem as a cluster-aware likelihood maximisation, show that it coincides with the incomplete-data maximum likelihood of a latent-variable mixture, and solve it by a generalised Expectation-Maximisation procedure whose otherwise intractable expert-quality term is supplied by an adaptively refined surrogate. We prove that the iterates converge whenever the surrogate errors are summable, and that at every limit point no candidate the search produces improves the true objective. On a heterogeneous image-classification mixture the method recovers the underlying domain partition on 95% of clusters without ever observing domain labels, and on that benchmark and a four-domain time-series forecasting one alike it outperforms the MoE and NAS baselines that likewise use no label information.

Petr Babkin, Oleg Bakhteev

Expectation-Maximization Neural Architecture Search Mixture-of-Experts Models

Jul 10, 2026cs.LG

Estimation, Prediction, and Assortment Optimization for Markov Chain Choice Models with Panel Data

We propose a framework for the Markov chain (MC) choice model with panel data, including parameter estimation, personalized choice prediction, and personalized assortment optimization. In contrast to the traditional setting, which assumes that each transaction is independently drawn from a random utility model, our framework accounts for dependencies among transactions for the same customer in historical data, captured by partial-ordering preference information. To the best of our knowledge, our framework initiates the study of choice modeling with panel data under MC. As our primary result, we propose novel expectation-maximization (EM) algorithms for MC parameter estimation by incorporating partial-ordering-based customer preference information. On synthetic datasets and the sushi dataset, our EM algorithms outperform the traditional EM algorithm of Simsek and Topaloglu (Operations Research, 66, 2018) and multinomial-logit-based partial-order benchmarks adapted from Jagabathula and Vulcano (Management Science, 64, 2018). As our secondary contribution, we present hardness and computational results for conditional choice prediction and assortment optimization problems. These results complement our estimation framework and clarify the computational landscape of conditional choice and assortment optimization, which may be of independent interest.

Yalcin Akcay, Gerardo Berbeglia, Young-San Lin

Melbourne Business School

Expectation-Maximization Discrete Choice Modeling Assortment Optimization

Jun 22, 2026astro-ph.IM

Stochastic Expectation Maximization for Robust State-Space Radio Interferometric Imaging

State--space models provide a flexible framework for analyzing dynamical systems, yet they often rely on Gaussian assumptions that fail to capture heavy-tailed or outlier-prone measurement noise. We propose a robust estimation scheme for linear state--space models subject to compound-Gaussian noise, as encountered for instance in radio interferometry affected by radio-frequency interference (RFI). The method relies on a Stochastic Approximation Expectation--Maximization (SAEM) algorithm in which the standard E-step is replaced by Monte Carlo sampling of the latent states and noise texture through closed-form Gibbs updates, enabling tractable inference despite the heavy-tailed likelihood. Numerical experiments show that the proposed method significantly improves reconstruction fidelity and robustness to RFI, outperforming a Gaussian EM algorithm and even an oracle RTS smoother. These results highlight the benefits of heavy-tailed state--space modeling and SAEM-based inference in interference-dominated imaging scenarios.

Nawel Arab, Mohammed Nabil El Korso, Isabelle Vin +1

†SATIE, ENS Paris-Saclay, Universit´e Paris-Saclay, France · ‡L2S, CentraleSupelec, Universit´e Paris-Saclay, France

Image Reconstruction Expectation-Maximization State Space Models

Jun 12, 2026cs.LG

EM-NeSy: Expectation Maximization for Neurosymbolic Learning

Neurosymbolic (NeSy) models integrate neural networks and symbolic reasoning for robust and interpretable AI. State-of-the-art NeSy models require that the symbolic component is expressed in a differentiable way, often complicating the use of approximate inference. We propose EM-NeSy which casts probabilistic NeSy learning as an instance of the Expectation-Maximization (EM) algorithm. In the expectation step, we compute the posterior over the neurally predicted symbols conditioned on the label via probabilistic inference. In the maximization step, we update the neural parameters based on this posterior using gradient descent only through the neural component. This formulation unlocks the full potential of the EM algorithm for NeSy learning. It allows NeSy to extend naturally to approximate reasoning without any additional modifications or differentiability requirements of the symbolic component. Furthermore, it recovers the standard end-to-end gradient-based NeSy setting under exact inference. Our experimental results demonstrate the scalability and computational efficiency of EM-NeSy.

Annegret Seibt, Luc De Raedt, Giuseppe Marra

Department of Computer Science KU Leuven

Symbolic Reasoning Expectation-Maximization Neuro-Symbolic Reasoning Neurosymbolic Learning

May 28, 2026cs.MS

libhmm: A Modern C++20 Library for Hidden Markov Models with Correct MLE Emission M-Steps

We describe libhmm, a C++20 library for Hidden Markov Model parameter estimation, sequence decoding, and model selection. libhmm addresses two gaps in existing software: the absence of a well-maintained, zero-dependency C++ HMM library suitable for embedding in production systems, and the widespread use of method-of-moments (MOM) approximations in the emission distribution M-step of the Baum-Welch algorithm. The library implements correct maximum likelihood estimators for sixteen scalar emission distributions, including an ECME algorithm for the location-scale Student-t distribution, Newton-Raphson maximization for Gamma, Beta, Weibull, and Negative Binomial distributions, and the von Mises distribution for circular data. All forward-backward and Viterbi calculations operate in full log-space. SIMD acceleration is provided for AVX-512, AVX2, SSE2, and ARM NEON via compile-time dispatch with scalar fallback. Version 4 adds multivariate observation support via the BasicHmm<Obs> template, with three multivariate emission families (diagonal Gaussian, full-covariance Gaussian, and independent components) each with correct weighted MLE M-steps. Python bindings are available via the companion package pylibhmm. We compare libhmm against established C and C++ HMM libraries and against published R reference packages on seven real-data benchmarks, and discuss the architectural tradeoffs made in the design.

Gary Wolfman

Independent Researcher

Parameter Estimation Markov Models Expectation-Maximization Sequence Modeling Hidden Markov Models

May 9, 2026cs.LG

Higher-Order Equilibrium Tracking for EM-Compressible Online Estimation

We study online estimation in latent-variable models by recasting the problem as tracking a moving empirical equilibrium. Standard online EM and stochastic approximation analyses primarily study convergence toward the population parameter and typically do not isolate the empirical batch optimum from the online tracking error at finite horizon. Our framework decomposes the online estimate into the frozen batch equilibrium at the current running statistic and a tracking lag that captures the algorithm's delay behind this moving target. We prove a batch-to-online transfer theorem: provided

\lVert e_T \rVert_{L^{2}} = o(T^{-1/2})

, the online estimator inherits the batch central limit theorem and the sharp first-order risk constant. Our key observation is that the empirical optimum evolves on a smooth equilibrium manifold indexed by the running statistic. An

m

-th order equilibrium-jet predictor combined with an order-

ν

frozen corrector yields localized tracking rates

O(T^{-ν(m+1)})

. We formalize EM-compressibility and EM-jet

^R

-compressibility as the structural conditions that make the equilibrium response and the Newton corrector evaluable from a retained streaming statistic. The theory is instantiated in latent linear Gaussian covariance estimation, where the first-order scheme operates on a compressed

d \times d

statistic with explicit finite-sample risk envelopes and a certified restart rule.

ZhiMing Li, Yue Song

School of Computer Science and Technology TianJin University Tianjin 300350, China · College of AI Tsinghua University Beijing 100084, China

Latent Variable Models Stochastic Approximation Expectation-Maximization

May 4, 2026stat.ML

Online Generalised Predictive Coding

This paper introduces an extension of generalised filtering for online applications. Generalised filtering refers to data assimilation schemes that jointly infer latent states, learn unknown model parameters, and estimate uncertainty in an integrated framework -- e.g., estimate state and observation noise -- at the same time (i.e., triple estimation). This framework appears across disciplines under different names, including variational Kalman-Bucy filtering in engineering, generalised predictive coding in neuroscience, and Dynamic Expectation Maximisation (DEM) in time-series analysis. Here, we specialise DEM for ``online'' data assimilation, through a separation of temporal scales. We describe the variational principles and procedures that allow one to assimilate data in a way that allows for a slow updating of parameters and precisions, which contextualise fast Bayesian belief updating about the dynamic hidden states. Using numerical studies, we demonstrate the validity of online DEM (ODEM) using a non-linear -- and potentially chaotic -- generative model, to show that the ODEM scheme can track the latent states of the generative process, even when its functional form differs fundamentally from the dynamics of the generative model. Framed from a neuro-mimetic predictive coding perspective, ODEM offers a biologically inspired solution to online inference, learning, and uncertainty estimation in dynamic environments.

Mehran H. Z. Bazargani, Szymon Urbas, Adeel Razi +2

Insight Research Ireland Centre, UCD, Dublin, Ireland. · UCD School of Mathematics and Statistics, UCD, Belfield, Dublin, Ireland. · Turner Institute for Brain and Mental Health, School of Psychological Sciences, Monash University, Melbourne, Victoria, Australia. +5

Parameter Estimation Latent Dynamics Modeling Predictive Coding Data Assimilation Expectation-Maximization

Apr 28, 2026stat.ME

Fitting Large Nonlinear Mixed Effects Models Using Variational Expectation Maximization

Nonlinear Mixed Effects (NLME) models are widely used in pharmacometrics and related fields to analyze hierarchical and longitudinal data. However, as the number of parameters and random effects increases, traditional methods for maximizing the marginal likelihood become computationally expensive. This paper explores the Variational Expectation Maximization (VEM) algorithm, a scalable alternative for fitting NLME models. Originally introduced in the context of probabilistic graphical models and later popularized through variational autoencoders, VEM has not been extensively applied to NLME modeling. By leveraging flexible variational families and reverse-mode automatic differentiation, VEM can efficiently maximize the marginal likelihood, scaling to NLME models with over 15,000 population parameters. This work provides a detailed description of VEM, compares it to other NLME fitting algorithms, and highlights its scalability through computational experiments. Using the Pumas statistical software, we fit two test models: 1) a standard warfarin model, and 2) an unnecessarily over-parameterized DeepNLME Friberg model with 15,410 population parameters and 16 random effects. The warfarin model was fitted to completion to demonstrate the correctness of VEM, while the DeepNLME Friberg model instead demonstrates VEM's scalability on a toy but large model. VEM improves the log likelihood steadily over hundreds of iterations at a practical per-iteration cost, while FOCE fails to complete even one iteration within a day. The model is deliberately over-parameterized for its small dataset and over-fits it, so what this experiment establishes is that VEM optimizes the objective of a model of this size at a practical cost. Applying VEM to large models that are genuinely useful is left to future work.

Mohamed Tarek, Pedro Afonso

PumasAI, USA

Expectation-Maximization Variational Inference Mixed-Effects Models

Apr 28, 2026stat.ME

Fractionally Supervised Classification with Maxima Nominated Samples

Fractionally supervised classification (FSC) offers a flexible framework for combining labeled and unlabeled data in model-based classification, but existing formulations assume simple random sampling. In many applications, however, the retained observation is an extreme order statistic from a set rather than a randomly selected unit. This is particularly appealing when the target population is rare, since maxima nomination sampling (NS) can enrich the sample with the most informative observations, as in screening, environmental monitoring, repeated testing, and reliability studies. Under such designs, the likelihood function changes fundamentally, and the usual FSC EM construction is no longer valid. We develop FSC for nominated samples by introducing a latent representation that accounts for both the class membership of the observed maximum and the latent composition of the remaining units in the set. The resulting method yields a proper EM algorithm and a coherent weighted-likelihood FSC procedure for NS data. We present the methodology in general form, illustrate it for a rare-event contamination normal mixtures, and show through simulation that it substantially improves on the misspecified alternative by ignoring the extra rank information of such data. A real-data analysis demonstrates its practical value.

Mohammad Jafari Jozani, Jingyu Wang

Department of Statistics, University of Manitoba, Winnipeg, MB, Canada · RBC Capital Markets, Toronto, Canada

Latent Variable Models Expectation-Maximization

Dec 31, 2025cs.LG

Gradient Descent as Implicit EM in Distance-Based Neural Models

Neural networks trained with standard objectives exhibit behaviors characteristic of probabilistic inference: soft clustering, prototype specialization, and Bayesian uncertainty tracking. These phenomena appear across architectures -- in attention mechanisms, classification heads, and energy-based models -- yet existing explanations often rely on loose analogies to mixture models or post-hoc architectural interpretation. We provide a direct explanation. For any objective with log-sum-exp structure over distances or energies, the gradient with respect to each distance is exactly the negative posterior responsibility of the corresponding component:

\partial L / \partial d_j = -r_j

. The identity is algebraic, requiring only differentiability; it is a specialization of Fisher's identity, and its significance here is its address: standard neural objectives instantiate it without modification. The consequence is that gradient descent on such objectives performs generalized expectation-maximization implicitly, with responsibilities arising as gradients to be applied rather than auxiliary variables to be computed. This result unifies three regimes of learning: unsupervised mixture modeling, where responsibilities are fully latent; attention, where responsibilities are conditioned on queries; and cross-entropy classification, where supervision clamps responsibilities to targets. Our claims live at training time: the responsibility-weighted gradient dynamics recently documented in transformers follow from the objective's geometry. The in-context Bayesian computation that trained transformers perform at inference time is the endpoint of these dynamics, not their per-step content.

Alan Oursland

Expectation-Maximization Neural Network Training Dynamics Gradient Descent Dynamics

Sep 2, 2025cs.LG

Differentiable Expectation-Maximisation and Applications to Gaussian Mixture Model Optimal Transport

The Expectation-Maximisation (EM) algorithm is a central tool in statistics and machine learning, widely used for latent-variable models such as Gaussian Mixture Models (GMMs). Despite its ubiquity, EM is typically treated as a non-differentiable black box, preventing its integration into modern learning pipelines where end-to-end gradient propagation is essential. In this work, we present and compare several differentiation strategies for EM, from full automatic differentiation to approximate methods, assessing their accuracy and computational efficiency. As a key application, we leverage this differentiable EM in the computation of the Mixture Wasserstein distance

\mathrm{MW}_2

between GMMs, allowing

\mathrm{MW}_2

to be used as a differentiable loss in imaging and machine learning tasks. To complement our practical use of

\mathrm{MW}_2

, we contribute a novel stability result which provides theoretical justification for the use of

\mathrm{MW}_2

with EM, and also introduce a novel unbalanced variant of

\mathrm{MW}_2

. Numerical experiments on barycentre computation, colour and style transfer, image generation, and texture synthesis illustrate the versatility of the proposed approach in different settings.

Samuel Boïté, Eloi Tanguy, Julie Delon +2

Université Paris Cité, CNRS, MAP5, F-75006 Paris, France · Centre Borelli, CNRS and ENS Paris-Saclay, F-91190 Gif-sur-Yvette, France · CMAP, CNRS, Ecole Polytechnique, Institut Polytechnique de Paris

Gaussian Mixture Models Expectation-Maximization Differentiable Optimization Optimal Transport Wasserstein Distance

Oct 1, 2024stat.ML

FastManly: An EM-Gradient Algorithm for Manly Mixture Models

A faster implementation of mixtures of Manly transformations is proposed. This method, called FastManly, uses Newton's method for optimization in an EM gradient algorithm instead of Nelder-Mead in a traditional EM. A gradient and full Hessian are derived. Simulations show improved performance with noticeable speedups.

Katharine M. Clark, Paul D. McNicholas

Department of Mathematics & Statistics, Trent University, Ontario, Canada. · Department of Mathematics & Statistics, McMaster University, Ontario, Canada.

Parameter Estimation Expectation-Maximization

Sep 16, 2024stat.ML

The EM-algorithm and the Method of Moments in Softmax Mixture Models

Softmax Mixture Models (SMMs) are discrete

K

-component mixture models for the probabilities of selecting one of

p

candidate feature vectors

X_1,\ldots,X_p\in\mathbb{R}^L

in heterogeneous populations and are widely used in econometrics and scientific applications. Related softmax mixture mechanisms also appear in modern LLM architectures. We provide a theoretical and methodological study of SMMs, focusing on the Expectation-Maximization (EM) algorithm and the Method of Moments (MoM). We show that EM recovers the mixture atoms at the parametric rate, up to logarithmic factors, after

\mathcal{O}(\log N)

iterations, provided atom separation is at least of order

\log K

. This improves on separation conditions in existing analyses of EM for high-dimensional Gaussian mixtures. We also develop MoM procedures for parameter and subspace estimation. Although MoM parameter estimates converge more slowly than EM and can deteriorate with

K

, they provide provable warm starts for EM and are useful for small

K

. For general

K

, we estimate the atom subspace via MoM and recommend running EM from multiple random initializations within this subspace. Finally, as

p\to\infty

, we show that SMMs approximate mixtures of exponential tilts of the feature distribution, yielding asymptotic identifiability.

Xin Bing, Florentina Bunea, Jonathan Niles-Weed +1

Department of Statistical Sciences, University of Toronto · Department of Statistics and Data Science, Cornell University · Courant Institute of Mathematical Sciences and Center for Data Science, New York University +1

Expectation-Maximization Discrete Choice Modeling Statistical Learning Theory

Jul 1, 2024cs.CV

An Expectation-Maximization Algorithm for Training Clean Diffusion Models from Corrupted Observations

Diffusion models excel in solving imaging inverse problems due to their ability to model complex image priors. However, their reliance on large, clean datasets for training limits their practical use where clean data is scarce. In this paper, we propose EMDiffusion, an expectation-maximization (EM) approach to train diffusion models from corrupted observations. Our method alternates between reconstructing clean images from corrupted data using a known diffusion model (E-step) and refining diffusion model weights based on these reconstructions (M-step). This iterative process leads the learned diffusion model to gradually converge to the true clean data distribution. We validate our method through extensive experiments on diverse computational imaging tasks, including random inpainting, denoising, and deblurring, achieving new state-of-the-art performance.

Weimin Bai, Yifei Wang, Wenzheng Chen +1

Academy for Advanced Interdisciplinary Studies, Peking University · College of Future Technology, Peking University · National Biomedical Imaging Center, Peking University +3

Expectation-Maximization Diffusion-Based Inverse Problems Inverse Problems Diffusion Models Image Denoising

Jul 14, 2019stat.ME

Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts

Mixtures of experts (MoE) are conditional mixture models in which both the mixing proportions and the component densities depend on the predictors, and are widely used for regression, classification and model-based clustering of heterogeneous data. Fitting MoE by maximum likelihood becomes unstable, and sometimes infeasible, when the predictors are numerous or correlated. We propose a regularized maximum likelihood framework for simultaneous parameter estimation and feature selection in MoE whose experts belong to the generalized linear model family, covering Gaussian, Poisson and multinomial responses within a single formulation. Sparsity is induced in both the gating network and the experts through

\ell_1

penalties, and the penalized log-likelihood is maximized by a proximal Newton-EM algorithm whose M-step reduces to weighted Lasso problems with closed-form coordinate-ascent updates. Unlike existing penalized MoE procedures, the algorithm requires neither a local quadratic approximation of the penalty nor any matrix inversion, it returns exactly sparse estimates without thresholding, and a proximal Newton-type variant guarantees a monotone increase of the penalized objective at every iteration. On simulated data and five real data sets, the method recovers the actual sparsity support and delivers prediction and clustering accuracy that is competitive with, and often better than, state-of-the-art regularized MoE. The source codes of our developed algorithms and their documentation are publicly available on Github at https://github.com/nv-thin/GLM-RMoE.

Thin Nguyen-Van, Faicel Chamroukhi, Ha Hoang Van +1

Faculty of Mathematics and Computer Science, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · IRT SystemX, 91120 Palaiseau, France +1

Mixture of Experts Sparse Mixture-of-Experts Expectation-Maximization Feature Selection Generalized Linear Models