cs.LGNov 12, 2026

Do Mixture-of-Experts Routers Benefit from Ensembling?

Authors: Viktor Veselý, Sebastian Jitaru

Organizations: University of Groningen

Abstract

Ensembles of neural networks have repeatedly been shown to outperform single networks. Ensemble theory attributes this gain to the accuracy of the individual members and the diversity among them. Mixture-of-experts models already combine many experts, but a single router decides how the experts are weighted for each input. Motivated by ensemble theory, we apply ensembling to the routing step: an ensemble of routers, trained jointly with the experts, proposes the expert weights. We systematically evaluate this approach using underparameterized networks on two image-classification and two tabular-regression tasks. We analyse each experiment with a performance-diversity decomposition of the router ensemble, which, to our knowledge, has not previously been developed for routers. We report four findings. First, more routers lower the validation loss. Second, training routers to explicitly maximize expert diversity improves the mean performance on three of the four tasks. Third, explicitly maximizing router diversity has no effect on performance. Fourth, combining router weights with the arithmetic mean works better than combining them with the geometric mean. Routing benefits mildly from ensembling, but this benefit is offset by the computational cost of training additional routers.

Figures & tables

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts

    May 12, 2026Sagi Ahrac, Noya Hochwald, Mor GevaExpert Load BalancingParameter-Free Mixture-of-Experts Routing

  2. Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Experts

    Mar 3, 2026Sanae Lotfi, Lucas Caccia, Alessandro Sordoni +2Language Model EnsemblesLanguage Model Merging