Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training
Organizations: Microsoft, India · Singapore Institute of Technology, Singapore · MIDAS Lab, Indian Institute of Technology Kanpur, Kanpur, India · TurboML ·
Abstract
Most parameters of a Mixture-of-Experts (MoE) model belong to its experts, and standard training keeps every expert gradient in GPU memory until the optimizer step. This memory grows with the number and size of the experts, so it rises as MoE models scale. We introduce Sonar-MoE, which updates each expert tensor during the backward pass and frees its gradient right away. It brings optimizer-in-backward from dense models to batched MoE expert tensors, so the GPU never holds more than one layer of expert gradients. When we fine-tune OLMoE-1B-7B and Granite-3.1-3B-A800M in bf16 with two expert-kernel backends, Sonar-MoE lowers peak memory by 19% to 22%. Its updates are bitwise identical to standard training without expert gradient clipping, and held-out loss stays within 0.003 of all baselines. Each step also takes 17% to 23% less time than standard training with global-norm clipping. Updating whole batched tensors rather than individual experts is what keeps Sonar-MoE fast, as it avoids a 20% to 72% slowdown while using the same memory. Sonar-MoE matches optimizer-in-backward over all parameters in memory and speed while still clipping the dense parameters.
Figures & tables
| Model | Backend | Method | Peak alloc. GB ( ) | Peak res. GB | Step ms ( ) | Tok/s | Held-out loss |
|---|---|---|---|---|---|---|---|
| OLMoE-1B-7B | SonicMoE | Standard (global clip) | 55.55 | 56.42 | 142.8 | 768 | 1.2577 |
| Standard (no expert clip) | 55.55 (0.0%) | 56.42 | 115.9 ( 18.8%) | 958 | 1.2591 | ||
| Optimizer-in-backward (all) | 43.35 ( 22.0%) | 44.24 | 114.4 ( 19.9%) | 970 | 1.2612 | ||
| Sonar-MoE | 43.57 ( 21.6%) | 44.24 | 116.1 ( 18.7%) | 948 | 1.2596 | ||
| ScatterMoE | Standard (global clip) | 55.55 | 57.17 | 160.2 | 688 | 1.2585 | |
| Standard (no expert clip) | 55.55 (0.0%) | 57.17 | 133.3 ( 16.7%) | 825 | 1.2601 |
| Checkpoint | ARC-C | HellaSwag | MMLU |
|---|---|---|---|
| Pretrained | 0.482 | 0.771 | 0.527 |
| Standard (global clip) | 0.532 | 0.771 | 0.521 |
| Sonar-MoE | 0.516 | 0.772 | 0.521 |
| Model | Hooks | Calls | Peak GB | Step ms | Held-out |
|---|---|---|---|---|---|
| OLMoE | batched | 32 | 43.57 | 116.1 | 1.2596 |
| OLMoE | per-expert | 2048 | 43.58 | 139.7 | 1.2600 |
| Granite | batched | 64 | 21.14 | 84.4 | 1.1616 |
| Granite | per-expert | 2560 | 21.37 | 145.5 | 1.1606 |
| Runs compared | Expert weights | Other weights |
| Same baseline, run twice | 0 | 6 |
| Sonar-MoE vs. standard (no expert clip) | 0 | 6 |
| Sonar-MoE : per-expert vs. batched hooks | 0 | 5 |
| Opt.-in-backward (all) vs. standard (no clip) | 0 | 0 |
| Standard: global clip vs. no expert clip | 3.04B (47%) | 91.5M (19%) |
| Method | Expert weights | Other weights |
|---|---|---|
| Standard (global clip) | 19.3% | 22.4% |
| Sonar-MoE | 19.3% | 22.5% |
| Runs under cap | |||||
| Setup | Alloc. | Res. | ms | 46 GB | 30.5 GB |
| Standard | 55.55 | 56.40 | 140.8 | OOM | – |
| Standard + GC | 55.48 | 56.16 | 155.6 | – | – |
| Sonar-MoE | 43.56 | 44.22 | 111.8 | ✓ | – |
| Sonar-MoE + GC | 43.23 | 44.03 | 130.1 | – | – |
| Opt.-in-bwd | 43.35 | 44.24 | 109.9 | ✓ | – |
| Configuration | Held-out loss |
|---|---|
| Standard (global clip) | 1.2577 |
| 8-bit AdamW | 1.2574 |
| 8-bit AdamW + Sonar-MoE | 1.2601 |
| Std. (global) | Std. (no exp.) | Sonar-MoE | Peak GB | |
|---|---|---|---|---|
| 1 | 143.4 | 117.3 | 116.4 | 55.55 / 43.57 |
| 2 | 217.8 | 191.9 | 229.4 | 57.21 / 44.31 |
| 4 | 368.5 | 341.7 | 451.7 | 57.21 / 44.31 |
| 8 | 665.8 | 639.6 | 895.6 | 57.21 / 44.32 |
Mixture-of-Experts Models
Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.
See all 488 papers