Do Mixture-of-Experts Routers Benefit from Ensembling?
Organizations: University of Groningen
Abstract
Ensembles of neural networks have repeatedly been shown to outperform single networks. Ensemble theory attributes this gain to the accuracy of the individual members and the diversity among them. Mixture-of-experts models already combine many experts, but a single router decides how the experts are weighted for each input. Motivated by ensemble theory, we apply ensembling to the routing step: an ensemble of routers, trained jointly with the experts, proposes the expert weights. We systematically evaluate this approach using underparameterized networks on two image-classification and two tabular-regression tasks. We analyse each experiment with a performance-diversity decomposition of the router ensemble, which, to our knowledge, has not previously been developed for routers. We report four findings. First, more routers lower the validation loss. Second, training routers to explicitly maximize expert diversity improves the mean performance on three of the four tasks. Third, explicitly maximizing router diversity has no effect on performance. Fourth, combining router weights with the arithmetic mean works better than combining them with the geometric mean. Routing benefits mildly from ensembling, but this benefit is offset by the computational cost of training additional routers.
Figures & tables
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| HYPERPARAMETER | VALUE |
|---|---|
| Kernel initialization | LeCun normal |
| Bias initialization | Zero |
| Hidden-layer activation | ReLU |
| Optimizer | Adam |
| Learning rate | |
| Weight-decay coefficient |
| VARIABLE | SET | NOTES |
| Number of experts ( ) | { } | |
| Number of routers ( ) | { } | : uniform weights; : MoE. |
| Experts’ base width ( ) | { } | Widths: or . |
| Routers’ base width ( ) | { } | Widths: or . |
| Routers’ aggregation method | {arithmetic mean, geometric mean} | See Section 3.6 |
| Expert diversity gradient to routers | , | Section 3.4 . |
Mixture-of-Experts Models
Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.
See all 489 papers