OOM: Offload-Optimized Full-Parameter MoE Training via Memory-Execution Co-Design
Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.
Abstract
Offloading model state to host memory and streaming layer weights let a single multi-GPU node train large mixture-of-experts (MoE) models. GPU memory then becomes a performance budget, shared by resident expert state, retained activations, and execution worksets. The best allocation depends on the expert execution path, batch size, and sequence length. We present OOM (Offload-Optimized MoE), a full-parameter MoE training system that co-designs expert execution and memory allocation. For expert-parallel execution, OOM updates resident experts on their owning GPUs and jointly plans state residency and MoE activation retention according to pipeline dependencies. For local execution, OOM bounds GPU worksets through dependency-aware partitioning; in hybrid models, it also retains attention results on the host to avoid costly recomputation and selectively regenerates layer boundaries through cheaper layers to make room. Calibrated cost models guide joint configuration within GPU and host budgets to minimize step time; OOM selects among the configured paths through short executions. On four H100s, OOM trains with 1M-token contexts, whereas FSDP2, ZeRO-Offload, and MegaTrain exhaust GPU memory at 512K. It also trains a 122B-parameter MoE model, on which all three exhaust host memory. On four 24 GB RTX 4090s, it trains 512K-token contexts and 1,024 1K-token sequences per step without gradient accumulation. Across matched-global-batch comparisons, OOM delivers 1.58x and 4.08x geometric-mean (up to 2.14x and 7.77x) throughput over CPU-offloaded FSDP2 on the two platforms while using the least host memory among the evaluated systems -- as little as 1/3.0 of FSDP2's.
Figures & tables
| Operator | Compute | Dependency |
|---|---|---|
| GDN mixing | Recurrent state across sequence segments | |
| Full attention | Causal token interactions within each head | |
| Expert FFN | Token-wise evaluation after routing |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| System | Parallelism / canonical state | Activation / transfer | Gradient processing |
|---|---|---|---|
| OOM | EP4: selected expert state on GPU; Local: host | Workload-specific checkpointing, retention, and partitioning; EP4 full and Local shard transfer | GPU reduction, or BF16 partials with FP32 host accumulation |
| FSDP2 | CPU-sharded state | Layer checkpointing and activation offload | FP32 reduce-scatter |
| ZeRO-Offload | ZeRO-3 with CPU parameter and optimizer offload | Layer checkpointing and activation offload | BF16 communication |
| Megatron-LM | TP2/SP, EP4/ETP1, DP2, PP1; CPU optimizer | Full recomputation | FP32 main gradients |
| MegaTrain | Layer streaming with CPU offload | Its streaming path with FA3 | Host optimizer |
| KT-FT | Routed experts on CPU (AMX); others through FSDP2 | Recompute; official full-parameter path | BF16 expert gradients, FP32 update |
| Model | Experts | Active | |
|---|---|---|---|
| Mixtral-8 7B | 8 | 2 | 4 |
| Gemma4-26B-A4B | 128 | 8 | 16 |
| Qwen3-30B-A3B | 128 | 8 | 16 |
| Qwen3.6-35B-A3B | 256 | 8 | 32 |
| Qwen3.5-122B-A10B | 256 | 8 | 32 |
| Batch | Path | State / activation | Step (s) |
|---|---|---|---|
| 16 | EP4 | R27/K0 | 3.776 |
| 32 | EP4 | R23/K0 | 5.020 |
| 64 | EP4 | R3/K4 | 9.302 |
| 128 | Local | Host state, checkpoint offload | 12.177 |
| 256 | Local | Host state, checkpoint offload | 22.305 |
| K0 | K1 | |||
|---|---|---|---|---|
| Resident | Time | Peak | Time | Peak |
| R17 | 6.048 | 51.3 | 5.733 | 55.0 |
| R21 | 5.376 | 60.3 | 5.331 | 64.1 |
| Arm | Time | GPU | Host |
|---|---|---|---|
| Default | Out of GPU memory | — | — |
| + Split | 260.02 | 43.4 | 731.3 |
| + Keep | 200.46 | 43.4 | 892.5 |
| + Replay | 206.00 | 45.4 | 732.5 |
| Length | A | B | 1+1 | 3+1 |
|---|---|---|---|---|
| 32K | 10.69/431 | 11.70/441 | 11.75/431 | 14.62/426 |
| 64K | 12.32/451 | 11.62/471 | 12.94/451 | 15.17/441 |
| 128K | 26.48/491 | 23.27/531 | 26.46/491 | 26.29/471 |
| 256K | 77.83/571 | 62.55/651 | 66.85/569 | 66.88/531 |
| 512K | 260.0/731 | 200.5/893 | 205.5/732 | 208.2/652 |
| Length | Global | Reuse | Sliding | MoE | /LSE | Freed |
|---|---|---|---|---|---|---|
| 32K | 0.160 | 0.004 | 0.005 | 0.015 | 0.50 | 0.86 |
| 64K | 0.821 | 0.008 | 0.013 | 0.030 | 1.00 | 1.72 |
| 128K | 3.904 | 0.021 | 0.021 | 0.071 | 2.01 | 3.44 |
| 256K | 17.164 | 0.039 | 0.042 | 0.147 | 4.02 | 6.88 |
| Workload | Plan | Pred. | Meas. | Error |
|---|---|---|---|---|
| H100, B32, 4K | EP4 R23 | 5.135 | 5.020 | |
| H100, B96, 4K | EP4 R1/K1 | 13.73 | 13.78 | |
| 4090, B384, 1K | Local, 32K chunks | 38.66 | 40.38 | |
| H100, 128K | B | 23.21 | 23.27 | |
| H100, 128K | 3+1 | 24.85 | 26.29 | |
| H100, 256K | B | 62.73 | 62.76 |
| Model | Batch | Plan | EP4 | Local | |
|---|---|---|---|---|---|
| Qwen3.6 | 4K | 48 | R15/K1 | 7.08/7.29 | 9.27/10.20 |
| Qwen3.6 | 4K | 192 | R1 | 25.91/26.72 | 17.09/17.51 |
| Qwen3.6 | 1K | 32 | R27 | 4.03/3.54 | 10.20/10.35 |
| Qwen3.6 † | 1K | 128 | R23 | 5.54/5.12 | 10.27/10.26 |
| Qwen3.6 | 1K | 256 | R19 | 8.54/9.29 | 9.55/10.08 |
| Mixtral | 1K | 32 | R15 | 6.26/6.31 | 11.62/12.21 |
Memory-Efficient Training
Memory-efficient training reduces the memory needed to optimize neural networks by compressing, recomputing, or relocating training state.
See all 100 papers