cs.LGNov 3, 2026

Provably Learning Mixtures of Linear Experts

Authors: Adrian Candocia, Aamod Varma, Mark Davenport

Organizations: School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA.

Abstract

Mixture-of-Experts (MoE) models are widely deployed in modern machine learning. However, few convergence guarantees exist for gradient-based training of MoEs: the objective is nonconvex, and even simple mixtures trap first-order methods in spurious local optima. We analyze a canonical gated mixture featuring linear experts and a gate induced by Gaussian clusters in the input space, and give, to our knowledge, the first global convergence guarantee for jointly training the gate and the experts of such a model by gradient EM on its likelihood. Given a logarithmic overparameterization relative to the true number of experts mm, gradient EM converges from randomly initialized gates and least-squares-initialized experts to any prescribed accuracy in polynomially many iterations, under a separation condition on the gate centers. Our analysis establishes a gradient-dominance inequality for the coupled dynamics by decoupling the loss into per-cluster subproblems. Empirical results validate the predicted mlog⁡mm\log m phase transition and suggest that the least-squares warm start required by our analysis is not needed in practice: randomly initialized experts perform nearly as well. We also find that leaving the mixing weights unconstrained concentrates routing onto the true number of experts, whereas load-balancing penalties spread it, so that the fitted model can be pruned to a fraction of its width without loss of accuracy.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. GaP-MoE : Explicit Expert Specialization in Mixture-of-Experts via Orthogonal Gradient Accumulation and Projection

    Oct 21, 2026Donghyun Jeong, Yoon-Yeong KimSparse Mixture-of-ExpertsMixture-of-Experts Models

  2. Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts

    Jul 14, 2019Thin Nguyen-Van, Faicel Chamroukhi, Ha Hoang Van +1Mixture of ExpertsSparse Mixture-of-Experts