Organizations: Microsoft, India · Singapore Institute of Technology, Singapore · MIDAS Lab, Indian Institute of Technology Kanpur, Kanpur, India · TurboML · Indraprastha Institute of Information Technology Delhi, New Delhi, India · Nanyang Technological University, Singapore
Most parameters of a Mixture-of-Experts (MoE) model belong to its experts, and standard training keeps every expert gradient in GPU memory until the optimizer step. This memory grows with the number and size of the experts, so it rises as MoE models scale. We introduce Sonar-MoE, which updates each expert tensor during the backward pass and frees its gradient right away. It brings optimizer-in-backward from dense models to batched MoE expert tensors, so the GPU never holds more than one layer of expert gradients. When we fine-tune OLMoE-1B-7B and Granite-3.1-3B-A800M in bf16 with two expert-kernel backends, Sonar-MoE lowers peak memory by 19% to 22%. Its updates are bitwise identical to standard training without expert gradient clipping, and held-out loss stays within 0.003 of all baselines. Each step also takes 17% to 23% less time than standard training with global-norm clipping. Updating whole batched tensors rather than individual experts is what keeps Sonar-MoE fast, as it avoids a 20% to 72% slowdown while using the same memory. Sonar-MoE matches optimizer-in-backward over all parameters in memory and speed while still clipping the dense parameters.
Figures & tables
Figure 1: Alpaca fine-tuning of OLMoE-1B-7B and Granite-3.1-3B-A800M for 2,000 steps with the SonicMoE and ScatterMoE backends, standard training with global-norm clipping (baseline) vs. Sonar-MoE . (a) Training loss on OLMoE; (b) peak GPU memory; (c) throughput averaged over steps 201–2,000.
Figure 2: The Sonar-MoE backward pass. Left: a per-expert layout launches 2E weight-gradient kernels per layer, while the batched layout computes all experts with one grouped GEMM per weight tensor. Middle: in standard training, expert gradients pile up layer by layer until the optimizer step, reaching Mfloor . Right: a post-accumulate hook updates each batched tensor and frees its gradient as soon as it is ready, so retained gradient memory stays at the scale of one layer.
Model
Backend
Method
Peak alloc. GB ( Δ )
Peak res. GB
Step ms ( Δ )
Tok/s
Held-out loss
OLMoE-1B-7B
SonicMoE
Standard (global clip)
55.55
56.42
142.8
768
1.2577
Standard (no expert clip)
55.55 (0.0%)
56.42
115.9 ( − 18.8%)
958
1.2591
Optimizer-in-backward (all)
43.35 ( − 22.0%)
44.24
114.4 ( − 19.9%)
970
1.2612
Sonar-MoE
43.57 ( − 21.6%)
44.24
116.1 ( − 18.7%)
948
1.2596
ScatterMoE
Standard (global clip)
55.55
57.17
160.2
688
1.2585
Standard (no expert clip)
55.55 (0.0%)
57.17
133.3 ( − 16.7%)
825
1.2601
Table 1: Alpaca fine-tuning for 2,000 steps with batch size 2 and seed 42 on two expert-kernel backends; every row uses the same data order. Δ is relative to standard training with global clipping on the same model and backend. Step time is the median over steps 201–2,000. Same-seed repeats vary by at most 1.4% in step time and 0.0016 in held-out loss. All rows of one model and backend ran on one GPU; the OLMoE SonicMoE rows ran on a different GPU from all other rows, so compare step times only within a model and backend. Bold marks the lowest memory and step time per model and backend.
Checkpoint
ARC-C
HellaSwag
MMLU
Pretrained
0.482
0.771
0.527
Standard (global clip)
0.532
0.771
0.521
Sonar-MoE
0.516
0.772
0.521
Table 2: Downstream accuracy on OLMoE after 2,000 steps (full test sets). Standard errors are 0.015 for ARC-Challenge and 0.004 for HellaSwag and MMLU in every row.
Figure 3: Where step time goes (ms per step): torch.profiler over steps 300–309. Profiling adds CPU overhead, so absolute times are higher than in Table 1 ; compare bars only. “Clip” covers the norm and scaling kernels of global-norm clipping, which take 1.8–1.9 ms on OLMoE and 1.1–1.2 ms on Granite when the expert gradients are not clipped. “Fwd+bwd” covers the forward and backward kernels.
Model
Hooks
Calls
Peak GB
Step ms
Held-out
OLMoE
batched
32
43.57
116.1
1.2596
OLMoE
per-expert
2048
43.58
139.7
1.2600
Granite
batched
64
21.14
84.4
1.1616
Granite
per-expert
2560
21.37
145.5
1.1606
Table 3: Hook granularity. Both layouts hold the same memory; per-expert hooks make 2EL optimizer calls per step instead of 2L .
Runs compared
Expert weights
Other weights
Same baseline, run twice
0
6
Sonar-MoE vs. standard (no expert clip)
0
6
Sonar-MoE : per-expert vs. batched hooks
0
5
Opt.-in-backward (all) vs. standard (no clip)
0
0
Standard: global clip vs. no expert clip
3.04B (47%)
91.5M (19%)
Table 4: Weights that differ between two runs after three steps on OLMoE (6.44B expert weights, 477M other weights).
Method
Expert weights
Other weights
Standard (global clip)
19.3%
22.4%
Sonar-MoE
19.3%
22.5%
Table 5: Share of weights that differ from their pretrained values after 2,000 steps of bf16 training on OLMoE.
Figure 4: Batch size on OLMoE. (a) Peak allocated memory (filled bars) and peak reserved memory (dashed outlines) for standard training and Sonar-MoE . The bold labels give the saving in allocated memory. (b) Step time in ms for each method. Sequences are padded to 256 tokens (512–8,192 tokens per step) so that kernel autotuning happens once, in warm-up.
Runs under cap
Setup
Alloc.
Res.
ms
46 GB
30.5 GB
Standard
55.55
56.40
140.8
OOM
–
Standard + GC
55.48
56.16
155.6
–
–
Sonar-MoE
43.56
44.22
111.8
✓
–
Sonar-MoE + GC
43.23
44.03
130.1
–
–
Opt.-in-bwd
43.35
44.24
109.9
✓
–
Table 6: Memory composition on OLMoE at batch size 2 (300-step runs). Peak allocated and reserved memory in GB, step time in ms, and whether 300 steps completed with the PyTorch allocator capped at 46 GB (standing for a 48 GB GPU) or 30.5 GB (a 32 GB GPU). GC = gradient checkpointing; opt.-in-bwd = all-parameter optimizer-in-backward.
Configuration
Held-out loss
Standard (global clip)
1.2577
8-bit AdamW
1.2574
8-bit AdamW + Sonar-MoE
1.2601
Table 7: Held-out loss on OLMoE after 2,000 steps with 8-bit AdamW, compared with standard training from Table 1 .
A
Std. (global)
Std. (no exp.)
Sonar-MoE
Peak GB
1
143.4
117.3
116.4
55.55 / 43.57
2
217.8
191.9
229.4
57.21 / 44.31
4
368.5
341.7
451.7
57.21 / 44.31
8
665.8
639.6
895.6
57.21 / 44.32
Table 8: Gradient accumulation with A microbatches per step (OLMoE, 500 steps). Step time in ms; peak allocated memory in GB for standard training (both variants) / Sonar-MoE . Sonar-MoE makes A updates per step.
Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.