Sparse Mixture-of-Experts

Follow this topic Ask about this topic

Sparse mixture-of-experts models use a router to send each token or input to only a few expert networks, aiming to combine large model capacity with manageable computation. Common approaches include top-k and threshold routing, hierarchical or shared experts, load-balancing objectives, and upcycling dense models into expert-based ones. Research spans language models as well as vision, speech, forecasting, and multimodal tasks, with work on training stability, pruning, quantization, and efficient inference. A central tension is that balancing expert use can prevent collapse but may weaken specialization, while high stored capacity does not always translate into better results or lower end-to-end cost.

Latest papers

121 papers16 papers in the last four weeks, up 33% on the four weeks before. 0.1% of all new papers.

Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts
Jul 14, 2019Thin Nguyen-Van, Faicel Chamroukhi, Ha Hoang Van +1Mixture of Experts Sparse Mixture-of-Experts
Faculty of Mathematics and Computer Science, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · IRT SystemX, 91120 Palaiseau, France +1

Jul 14, 2019J/K move · Enter open · S save

Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts

Thin Nguyen-Van, Faicel Chamroukhi, Ha Hoang Van, Bao Tuyen Huynh

Faculty of Mathematics and Computer Science, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · IRT SystemX, 91120 Palaiseau, France+1

Mixture of Experts Sparse Mixture-of-Experts Expectation-Maximization Feature Selection Generalized Linear Models

Open paper AskPDF arXiv

Mixtures of experts (MoE) are conditional mixture models in which both the mixing proportions and the component densities depend on the predictors, and are widely used for regression, classification and model-based clustering of heterogeneous data. Fitting MoE by maximum likelihood becomes unstable, and sometimes infeasible, when the predictors are numerous or correlated. We propose a regularized maximum likelihood framework for simultaneous parameter estimation and feature selection in MoE whose experts belong to the generalized linear model family, covering Gaussian, Poisson and multinomial responses within a single formulation. Sparsity is induced in both the gating network and the experts through

\ell_1

penalties, and the penalized log-likelihood is maximized by a proximal Newton-EM algorithm whose M-step reduces to weighted Lasso problems with closed-form coordinate-ascent updates. Unlike existing penalized MoE procedures, the algorithm requires neither a local quadratic approximation of the penalty nor any matrix inversion, it returns exactly sparse estimates without thresholding, and a proximal Newton-type variant guarantees a monotone increase of the penalized objective at every iteration. On simulated data and five real data sets, the method recovers the actual sparsity support and delivers prediction and clustering accuracy that is competitive with, and often better than, state-of-the-art regularized MoE. The source codes of our developed algorithms and their documentation are publicly available on Github at https://github.com/nv-thin/GLM-RMoE.