Rethinking MoE Load Balancing as Feedback Control
Organizations: Department of Mathematics and Mathematical Statistics, Ume˚a University, Sweden
Abstract
Mixture-of-Experts (MoE) training is often not stationary: the data mixture changes at curriculum boundaries, when a model is continually pretrained into a new domain, or when the corpus is refreshed. Routing is learned against the current mixture, so each such change disrupts the per-expert load the router had converged to, and existing balancing methods have only been evaluated on stationary data. We rethink load balancing as a feedback control problem, in which the deviation of each expert's load from its fair share is the control error and the per-expert routing bias is the control input, and define a family of controllers, the PID-Balancer, from the proportional, integral, and derivative terms of classical PID control. Two widely used methods belong to this family: the auxiliary balancing loss behaves like a proportional controller, and the loss-free bias update is an integral controller acting on the sign of the error. Reading the router as a static map with a one-step delay, we show that the integral term is the only one that both removes persistent imbalance and averages out the step-to-step fluctuation of the measured load, whereas the proportional and derivative terms pass or amplify that fluctuation into the routing bias. We therefore adopt PID (I), the pure-integral member of the family with a single gain. Two settings are evaluated: (i) training an OLMoE backbone from scratch with 16 to 64 experts per layer, through a mid-run mixture switch, and (ii) continual pretraining of the released OLMoE-1B-7B on a new corpus. PID (I) holds balance before the shift and restores it more tightly afterwards than either the auxiliary loss or loss-free balancing, at comparable perplexity.
Figures & tables
| 1 1 | 6 1 | 7 1 | ||||||||
| E | Method | postMV | PPL | DER / UTR | postMV | PPL | DER / UTR | postMV | PPL | DER / UTR |
| 16 | Vanilla | 4.44 | 3.29 | 7.42% / 38.67% | 4.89 | 3.22 | 5.08% / 41.41% | 4.77 | 3.21 | 5.86% / 37.89% |
| Aux-loss | 0.00% / 0.23% | 0.00% / 0.00% | 0.00% / 0.36% | |||||||
| ALF-LB | 0.00% / 0.00% | 0.00% / 0.00% | 0.00% / 0.00% | |||||||
| PID (I) | 0.00% / 0.00% | 0.00% / 0.00% | 0.00% / 0.00% | |||||||
| 32 | Vanilla | 5.13 | 3.31 | 12.30% / 44.34% | 6.23 | 3.21 | 9.96% / 51.76% | 5.42 | 3.21 | 9.18% / 42.58% |
| Method | Setting | preMV | postMV | DER / UTR | PPL-CPT | PPL-Annealing | MMLU | HellaSwag | ARC-C | HumanEval | MBPP |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Base | - | - | - | - | 2.40 | 5.46 | 0.55 | 0.79 | 0.57 | 0.11 | 0.23 |
| Vanilla | cold | - | 5.83 | 1.07% / 29.10% | 2.19 | 7.23 | 0.50 | 0.74 | 0.50 | 0.12 | 0.18 |
| Aux-loss | cold | - | 4.43 | 0.00% / 1.56% | 2.19 | 7.21 | 0.49 | 0.74 | 0.50 | 0.12 | 0.18 |
| ALF-LB | cold | - | 1.01 | 0.00% / 0.00% | 2.19 | 7.36 | 0.46 | 0.71 | 0.48 | 0.10 | 0.14 |
| PID (I) | cold | - | 0.90 | 0.00% / 0.00% | 2.19 | 7.39 | 0.46 | 0.72 | 0.50 | 0.12 | 0.14 |
| Vanilla | warm | 1.14 | 5.76 | 0.59% / 27.73% | 2.24 | 6.07 | 0.51 | 0.77 | 0.55 | 0.12 | 0.18 |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| OLMoE-16 | OLMoE-32 | OLMoE-64 | OLMoE-1B-7B | |
| Backbone | ||||
| Hidden size | 1280 | 1280 | 1280 | 2048 |
| Layers (all MoE) | 16 | 16 | 16 | 16 |
| Attention heads | 20 | 20 | 20 | 16 |
| Head dim | 64 | 64 | 64 | 128 |
| Normalisation | RMSNorm, 1e-5 | RMSNorm, 1e-5 | RMSNorm, 1e-5 | RMSNorm, 1e-5 |
| Pretraining | Continual pretraining | |
|---|---|---|
| Corpus | ||
| Corpus | SlimPajama-627B | Anneal: Dolmino CPT: codeparrot-clean |
| Domains | 7 domains (arxiv, Books, c4, commoncrawl, GitHub, stackexchange, wikipedia) | Anneal: 6 domains (dclm, flan, math, pes2o, stackexchange, wiki). CPT: 1 domain (code) |
| Tokenizer | gpt-neox-olmo-dolma-v1_5 | OLMoE-native tokenizer |
| Vocabulary / EOS / BOS | 50,280 (embedding padded to 50,304), EOS 50279, no BOS | 50,280 (embedding padded to 50,304), EOS 50279, no BOS |
| Training consumption | ||
| Method | Coefficient | Value | DER | UTR |
|---|---|---|---|---|
| Aux-loss | 1e-5 | 15.14% | 44.73% | |
| 1e-4 | 16.21% | 43.26% | ||
| 1e-3 | 3.61% | 24.02% | ||
| 1e-2* | 0.06% | 0.21% | ||
| 1e-1 | 0.00% | 0.00% | ||
| ALF-LB | 1e-5 | 0.00% | 0.49% |
| Routing sparsity | |||
|---|---|---|---|
| Setpoint | 2048 | 4096 | 8192 |
| Ceiling | 15 | 7 | 3 |
| Method | Gains | PreMV | PostMV | RMS | DER | Active | Flip |
|---|---|---|---|---|---|---|---|
| Vanilla | – | 5.08 | 6.08 | – | 16.3% | 94.3% | – |
| PID (I) | 1e-4 | 0.62 | 0.40 | 4.0e-2 | 0.0% | 100.0% | 37.6% |
| PID (P) | 1e-4 | 5.17 | 5.62 | 1.6e-4 | 15.0% | 92.8% | 67.7% |
| PID (P) | 1e-3 | 4.35 | 5.10 | 1.4e-3 | 10.7% | 96.8% | 69.2% |
| PID (P) | 1e-2 | 2.72 | 2.86 | 8.6e-3 | 1.2% | 98.2% | 84.5% |
| PID (P) | 1e-1 | 5.22 | 5.33 | 1.6e-1 | 5.6% | 79.1% | 94.6% |
| Method | Gains | PreMV | PostMV | RMS | DER | Active | Flip |
|---|---|---|---|---|---|---|---|
| PID (I) | 0.62 | 0.40 | 4.0e-2 | 0.0% | 100.0% | 37.6% | |
| PID (PI) | 0.81 | 0.46 | 4.1e-2 | 0.0% | 100.0% | 58.6% | |
| PID (PI) | 1.23 | 0.62 | 3.4e-2 | 0.0% | 99.9% | 70.2% | |
| PID (PI) | 2.79 | 1.84 | 3.2e-2 | 0.0% | 98.7% | 83.6% | |
| PID (DI) | 1.06 | 0.52 | 4.1e-2 | 0.0% | 100.0% | 69.1% | |
| PID (DI) | 2.58 | 1.47 | 3.9e-2 | 0.0% | 99.9% | 77.9% |
| mean | fluctuation | |||||||
| Method | Gains | I | P | D | I | P | D | |
| PID (I) | 4.0e-2 | – | – | 7.8e-4 | – | – | 0 | |
| PID (P) | – | 7.9e-3 | – | – | 3.6e-3 | – | – | |
| PID (D) | – | – | 1.3e-7 | – | – | 4.5e-4 | – | |
| PID (PI) | 4.1e-2 | 1.4e-6 | – | 9.3e-4 | 1.8e-5 | – | 1.5 | |
| PID (PI) | 3.4e-2 | 9.3e-6 | – | 7.4e-4 | 2.5e-4 | – | 17.5 | |
| Setting | preMV | postMV | DER / UTR | PPL-CPT | PPL-Annealing | MMLU | HellaSwag | ARC-C | HumanEval | MBPP | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.01 | cold | - | 4.43 | 0.00% / 1.56% | 2.19 | 7.21 | 0.49 | 0.74 | 0.50 | 0.12 | 0.18 |
| 0.08 | cold | - | 2.30 | 0.00% / 0.00% | 2.22 | 7.32 | 0.46 | 0.73 | 0.50 | 0.11 | 0.20 |
| 0.4 | cold | - | 0.95 | 0.00% / 0.00% | 2.25 | 7.60 | 0.46 | 0.72 | 0.49 | 0.11 | 0.14 |
| 0.8 | cold | - | 0.60 | 0.00% / 0.00% | 2.27 | 7.77 | 0.42 | 0.71 | 0.51 | 0.11 | 0.14 |
| 4 | cold | - | 0.23 | 0.00% / 0.00% | 2.29 | 8.38 | 0.41 | 0.71 | 0.51 | 0.10 | 0.13 |
| PID (I) | cold | - | 0.90 | 0.00% / 0.00% | 2.19 | 7.39 | 0.46 | 0.72 | 0.50 | 0.12 | 0.14 |
| preMV | postMV | PPL-CPT | PPL-Annealing | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Setting | Method | ||||||||||||
| cold | Vanilla | – | – | – | 5.83 | 5.82 | 5.76 | 2.19 | 2.11 | 2.02 | 7.23 | 7.33 | 7.53 |
| Aux-loss | – | – | – | 4.43 | 4.22 | 4.04 | 2.19 | 2.11 | 2.02 | 7.21 | 7.34 | 7.53 | |
| ALF-LB | – | – | – | 1.01 | 1.05 | 0.97 | 2.19 | 2.14 | 2.05 | 7.36 | 7.98 | 8.18 | |
| PID (I) | – | – | – | 0.90 | 0.94 | 0.85 | 2.19 | 2.13 | 2.03 | 7.39 | 7.98 | 8.19 | |
| warm | Vanilla | 1.14 | 1.14 | 1.14 | 5.76 | 5.77 | 5.79 | 2.24 | 2.18 | 2.10 | 6.07 | 6.15 | 6.28 |
| DER | UTR | ||||||
|---|---|---|---|---|---|---|---|
| Setting | Method | ||||||
| cold | Vanilla | 1.07% | 1.07% | 1.27% | 29.10% | 29.30% | 29.30% |
| Aux-loss | 0.00% | 0.00% | 0.00% | 1.56% | 1.27% | 0.59% | |
| ALF-LB | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | |
| PID (I) | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | |
| warm | Vanilla | 0.59% | 0.78% | 0.68% | 27.73% | 29.00% | 31.15% |
| MMLU | HellaSwag | ARC-C | HumanEval | MBPP | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Setting | |||||||||||||||
| Vanilla | cold | 0.50 | 0.50 | 0.48 | 0.74 | 0.73 | 0.73 | 0.50 | 0.49 | 0.51 | 0.12 | 0.14 | 0.14 | 0.18 | 0.22 | 0.19 |
| Aux-loss | cold | 0.49 | 0.50 | 0.48 | 0.74 | 0.73 | 0.73 | 0.50 | 0.50 | 0.51 | 0.12 | 0.12 | 0.14 | 0.18 | 0.21 | 0.18 |
| ALF-LB | cold | 0.46 | 0.47 | 0.46 | 0.71 | 0.71 | 0.70 | 0.48 | 0.49 | 0.49 | 0.10 | 0.11 | 0.10 | 0.14 | 0.12 | 0.16 |
| PID (I) | cold | 0.46 | 0.45 | 0.44 | 0.72 | 0.71 | 0.70 | 0.50 | 0.49 | 0.49 | 0.12 | 0.12 | 0.13 | 0.14 | 0.16 | 0.16 |
| Vanilla | warm | 0.51 | 0.51 | 0.50 | 0.77 | 0.76 | 0.76 | 0.55 | 0.55 | 0.55 | 0.12 | 0.12 | 0.10 | 0.18 | 0.20 | 0.20 |
| Annealing | CPT, cold start | CPT, warm start | ||||
| Method | MaxVio | MaxVio | MaxVio | |||
| Base | 0.83 | 0.141 | 5.23 | 0.141 | 5.23 | 0.141 |
| Vanilla | 0.91 | 0.146 | 5.79 | 0.137 | 5.71 | 0.143 |
| Aux-loss | 0.88 | 0.135 | 4.29 | 0.110 | 4.42 | 0.114 |
| ALF-LB | 0.34 | 0.145 | 0.46 | 0.074 | 0.51 | 0.077 |
| bias removed | 0.87 | 0.147 | 4.14 | 0.110 | 4.61 | 0.125 |
| PPL-CPT | PPL-Annealing | ||||||
| Method | Scenario | on | off | on | off | ||
| Base | – | 2.40 | – | 5.46 | – | ||
| Vanilla | cold | 2.19 | – | 7.23 | – | ||
| ALF-LB | cold | 2.19 | 2.17 | 7.36 | 7.17 | ||
| PID (I) | cold | 2.19 | 2.17 | 7.39 | 7.20 | ||
| Vanilla | warm | 2.24 | – | 6.07 | – | ||
Expert Load Balancing
Expert load balancing distributes tokens and compute across mixture-of-experts models to improve utilization without erasing expert specialization.
See all 44 papers