Scaling Mixture-of-Experts (MoE) training introduces systems challenges absent in dense models. Because each token activates only a subset of experts, this sparsity allows total parameters to grow much faster than per-token computation, creating coupled constraints across memory, communication, and computation. Optimizing one dimension often shifts pressure to another, demanding co-design across the full system stack. We address these challenges for MoE training through integrated optimizations spanning memory (fine-grained recomputation, offloading, etc.), communication (optimized dispatchers, overlapping, etc.), and computation (Grouped GEMM, fusions, CUDA Graphs, etc.). The framework also provides Parallel Folding for flexible multi-dimensional parallelism, low-precision training support for FP8 and NVFP4, and efficient long-context training. On NVIDIA GB300 and GB200, it achieves 1,233/1,048 TFLOPS/GPU for DeepSeek-V3-685B and 974/919 TFLOPS/GPU for Qwen3-235B. As a performant, scalable, and production-ready open-source solution, it has been used across academia and industry for training MoE models ranging from billions to trillions of parameters on clusters scaling up to thousands of GPUs. This report explains how these techniques work, their trade-offs, and their interactions at the systems level, providing practical guidance for scaling MoE models with Megatron Core.
Figures & tables
Figure 1
Figure 1: Data flow through an MoE layer: Route, Dispatch, Compute, and Combine stages.
Figure 2: Router architecture: linear projection, score function, top- k selection, and load balancing.
Component
Groups Used
Reason
Router
tp , cp , tp_cp
Weights duplicated across EP ranks
Token Dispatcher
ep , tp_ep
all-to-all across expert ranks
Experts
ep , expt_tp , expt_dp
Sharded across EP; gradients reduced in EDP
Shared Experts
tp
Same as dense MLP
Table 1: MoE component to process group mapping.
Figure 3: Dense Model vs MoE Model parameter/compute scaling.
Figure 4: Expert Parallelism (EP) distributes experts across GPUs. The all-to-all communication dispatches tokens to their assigned experts and combines results.
Aspect
Attention (Dense)
MoE (Sparse)
Computation
Every token attends to all others
Each token routes to K of E experts
TP
Large QKV matrices benefit from high TP
Small per-expert dimensions make high TP counterproductive
CP
Long sequences benefit from high CP
No sequence dependency; CP is irrelevant
EP
Not applicable (no experts)
Essential for distributing many experts
Table 2: Contrasting parallelism requirements of attention and MoE layers within a single Transformer block.
Figure 5: Parallelism mappings: traditional constraints vs. MoE Parallel Folding decoupling.
Figure 6: Parallel Folding: decoupled attention and MoE parallelism mappings.
Component
Memory per GPU
Optimization Techniques
Weights & Gradients
36.4 GB
PP, EP, or TP sharding
Main Weights & Optimizer States
32.1 GB
Distributed optimizer, BF16 moments
Activations
131.0 GB
Low Precision, Recomputation, Offloading
Total
199.5 GB
Table 3: Memory breakdown per GPU for DeepSeek-V3 with BF16 training ( PP4×VPP4×EP64 , 256 GPUs).
Figure 7: Memory-Efficient Permutation.
Figure 8: Selective Recomputation.
Recomputation Target
Memory Saved per GPU
MLA Up-Projection
30.4 GB
Activation Function (SwiGLU)
3.8 GB
LayerNorm
8.2 GB
Total
42.4 GB
Table 4: Memory reduction per GPU from fine-grained recomputation for DeepSeek-V3 ( PP4×VPP4×EP64 , 256 GPUs).
Figure 9: Fine-grained activation offloading: stream overlap for forward and backward passes.
Model & Config
Baseline
+Offload
Mem Δ
Throughput Δ
DeepSeek-V3 full
169 GB
151 GB
− 10.7%
− 1.6%
TP1PP8EP32VPP4, MXFP8
945 TF/s
930 TF/s
Qwen3-235B
172 GB
175 GB
+ 1.7%
+ 15.0%
TP2 → TP1 + EP16 → EP64
800 TF/s
920 TF/s
Table 5: Memory and throughput impact of fine-grained activation offloading.
Figure 10: Fine-grained offloading and recomputation: complementary memory optimization strategies.
Figure 11: Comparison of sharding strategies: (a) FSDP2 shards each parameter uniformly; (b) Megatron-FSDP flattens per-module and shards non-uniformly, aligning with communication buffers.
Figure 12: Persistent double-buffer design: two pre-allocated buffers are cycled across FSDP collectives, eliminating allocation overhead and enabling NCCL User Buffer Registration.
Technique
Memory Target
Trade-off
Reduced-Precision Training
Activations
Numerical Precision and CPU Overhead
Memory-Efficient Permutation
Activations
-
Fine-grained Recomputation
Activations
Compute Overhead
Fine-grained Offloading
Activations
CPU Overhead and Non-Overlapped Copy
Precision-aware Optimizer
Optimizer States
Numerical Precision
FSDP (with EP)
Params + Optimizer
Communication Overhead
Table 6: Summary of memory optimization techniques.
Figure 13: Expert parallelism across 4 GPUs with 4 experts.
Figure 14: The dispatch kernel design of HybridEP.
Figure 15: The combine kernel design of HybridEP.
GB200 (µs)
H100 (µs)
EP size
HybridEP
all-to-all
HybridEP
all-to-all
dispatch
8
391
735
661
1265
16
578
743
1485
5774
32
612
769
3064
8059
64
675
930
4626
9164
combine
8
353
741
624
1277
Table 7: EP Scaling Performance for HybridEP and all-to-all (in µs).
Figure 16: Merged FWD-FWD Timeline with all-to-all Overlapping.
Figure 17: Merged FWD-BWD Timeline with all-to-all Overlapping.
Figure 18: EP all-to-all communication overlap strategies: baseline vs. 1F1B with W/D split.
Figure 19: Interleaved PP Timeline with all-to-all Overlapping.
Figure 20: The pipeline for permute fusion in the training process.
Figure 21: The workflow of the router fusion.
Figure 22: Traditional execution (top) versus CUDA Graph execution (bottom).
Figure 23: Full versus layer-wise CUDA Graphs in one training iteration (three layers, two microbatches).
Figure 24: Partial CUDA Graphs capture static components (attention, shared experts, router, preprocessing) while leaving dynamic expert computation outside the graph.
Figure 25: Transformer layer forward pass: without (upper) and with (lower) partial CUDA Graphs. CPU overhead is largely eliminated for static components.
Figure 26: Why Pipeline Parallelism prevents CUDA Graphs from being shared across microbatches. With PP (top): Execution is interleaved—multiple forward passes run before any backward pass. If microbatches share a graph, F_mb1 overwrites saved context of F_mb0 before B_mb0 uses it, causing memory corruption. Each microbatch needs its own graph ( L×M×2 graphs total). Without PP (bottom): Execution is sequential—each microbatch completes (forward+backward) before the next starts. Context is consumed before being overwritten, so microbatches can safely share graphs (only L×2 graphs total).
Figure 27: ECHO workflow for forward and backward passes. The planner generates routing and hot expert maps. Expert Dispatch clones hot expert weights to spare slots; Expert Gradient Dispatch reduces gradients back to home experts.
Figure 28: Memory layout comparison across three execution modes. Left : Eager mode allocates memory dynamically based on actual usage. Middle : Baseline static shape requires worst-case sized buffers for each layer independently, causing severe fragmentation when actual usage is lower. Right : Paged Stashing uses a single worst-case tmp buffer shared across layers for computation, while a paged stashing buffer stores only the actual tokens, significantly reducing total memory footprint.
Figure 29: Paged Stashing stream overlap. Forward pass : After Layer N computes, its activations are stashed (copied from tmp buffer to paged stashing buffer) on a dedicated Pack stream while Layer N+1 computes on the main Compute stream—the stash is completely overlapped. Backward pass : Activations for Layer N are pre-fetched (reloaded from stashing buffer to tmp buffer) on the Unpack stream before Layer N backward starts, hiding reload latency.
Wall
Reduced-Precision Training Benefit
Details
Memory
50% (FP8)/75% (FP4) activation reduction
Section 4.1.3
Eliminate BF16 weight copies
Section 5.3.5
BF16 optimizer states
Section 4.1.6
Communication
50% parameter AllGather 1
Section 5.3.5
Compute
Faster Tensor Core GEMMs
Section 4.3.5
Quantization kernel overhead
Section 4.3.2
Table 8: Summary of the impact from reduced-precision training on the Three Walls.
Figure 30: FP8 training recipes: Per-Tensor Scaling, Blockwise FP8, and MXFP8.
Figure 31: The computation of a linear layer with various FP8 recipes. Note the differences in quantization granularity and tensor layout requirements across platforms.
Figure 32: FP8 primary weight quantization scheme for blockwise scaling.
Figure 33: FP8 primary weight quantization scheme for delayed scaling and per-tensor current scaling.
Figure 34: SDPA exhibits O(s2) complexity, while MoE and the remaining attention operations exhibit O(s) complexity. Therefore, SDPA dominates the computation at longer sequence lengths.
Platform
Sequence Length
Forward TFLOPS
Backward TFLOPS
Hopper
4096
553
422
16384
638
523
Blackwell
4096
1324
1083
16384
1698
1298
Table 9: SDPA performance in cuDNN for DeepSeek-V3.
Figure 35: Communication and computation patterns of TP and two types of CP.
Figure 36: Unpacked vs. Packed sequences.
Figure 37: Compute imbalance in causal attention over packed sequences.
Figure 38: Dynamic Context Parallelism for Packed Sequences.
Figure 39: Load balancing strategies in Megatron-Core MoE.
Figure 40: Shared expert architecture in Megatron-Core MoE. The shared expert processes all tokens while routed experts process only their assigned tokens. When overlap is enabled, shared expert computation runs in parallel with the token dispatch/combine communication, hiding its latency.
PP rank
VPP rank 0
VPP rank 1
0
embedding + 3 × decoder
2 × decoder
1–13
2 × decoder
2 × decoder
14
2 × decoder
MTP
15
2 × decoder
loss
Table 10: Layer distribution for DeepSeek-V3 with flexible asymmetric VPP ( PP=16 , VPP=2 ).
Figure 41: Flexible Pipeline Parallel Placement.
Figure 42: An example of granular upcycling a dense layer into E2G2T2 fine-grained MoE. E2G2T2 denotes 4 experts, top 2, with half intermediate size. (1) We shard MLP weights in the intermediate dimension ( 4h→2h ) then duplicate the shards. (2) We initialize half the router weights then duplicate them. This ensures Top2 always selects one of each MLP shard so MoE output is the same as the dense model at the start of training.
Model
System
#GPUs
Seqlen
Dtype
Per-GPU TF
Tokens/s/GPU
DeepSeek-V3
GB300
256
4,096
MXFP8
1233
4,730
DeepSeek-V3
GB200
256
4,096
MXFP8
1048
4,020
DeepSeek-V3
GB200
256
4,096
BF16
857
3,298
DeepSeek-V3
H100
1024
4,096
FP8-BLK
368
1,412
Qwen3-235B
GB300
256
4,096
MXFP8
974
6,583
Qwen3-235B
GB200
256
4,096
MXFP8
919
6,212
Table 11: Unified throughput benchmarks (per-GPU figures) for two mixture-of-experts models on NVIDIA GB300, GB200, and H100. All configurations use force-balanced routing. The Dtype column specifies the FP8 recipe: FP8-BLK denotes blockwise FP8 on Hopper, and MXFP8 denotes microscaling FP8 on Blackwell.
Strategy
Peak Activation
Weight Memory
Optimizer States
Comm (Per-Layer)
TP
1/d (with SP)
1/d
1/d
High
EP
∼ 1 (load-dependent)
1/d (MoE only)
1/d
Medium
PP
1 ( > 1 with VPP)
1/d
1/d
Medium
CP
1/d
1
1/d†
Medium
DP
1
1
1/d†
Low
Table 12: Impact of parallelism strategies on memory and communication. d = parallelism degree. † Requires distributed optimizer ( --use-distributed-optimizer ).
Table 17: DeepSeek-V3 final optimized configurations on GB200 and H100. † Parallel Folding is used; TP applies to the non-MoE modules only, expert TP is always 1.
Category
GB200
H100
Parallelism
Parallel Folding Flexible VPP
Parallel Folding Flexible VPP
Precision
MXFP8
FP8-Blockwise
Memory
Memory-efficient permutation Fine-grained recomputation FP8 primary weights Low-precision optimizer states Optimizer states offloading
Memory-efficient permutation Fine-grained recomputation FP8 primary weights Low-precision optimizer states
Communication
HybridEP
DeepEP EP Communication overlap
Compute Efficiency
Kernel fusions CUDA Graphs CPU-side optimizations
Kernel fusions
Table 18: DeepSeek-V3 optimization summary by platform.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Description
Model parameters
E
Number of experts in an MoE layer
K
Top- k routing width (experts activated per token; K≪E )
h
Hidden dimension
N
Total model parameters
Nactive
Parameters activated per token (scales with K )
Appendix
Table 19: Notation and abbreviations used throughout this report.
Model
Configuration
Hyper-parameters
DeepSeek-V3
GB300, 256 GPUs, 4k seqlen, MXFP8, 1233 TF
TP1 PP4 CP1 EP64 VPP4 MBS1 GBS8192
DeepSeek-V3
GB200, 256 GPUs, 4k seqlen, MXFP8, 1048 TF
TP1 PP4 CP1 EP64 VPP4 MBS1 GBS8192
DeepSeek-V3
GB200, 256 GPUs, 4k seqlen, BF16, 857 TF
TP1 PP8 CP1 EP32 VPP4 MBS1 GBS4096
DeepSeek-V3
H100, 1,024 GPUs, 4k seqlen, FP8-BLK, 368 TF
TP2 PP4 CP1 EP64 VPP4 MBS1 GBS8192
Qwen3-235B
GB300, 256 GPUs, 4k seqlen, MXFP8, 974 TF
TP1 PP4 CP1 EP64 VPP 0 6 MBS2 GBS3072
Qwen3-235B
GB200, 256 GPUs, 4k seqlen, MXFP8, 919 TF
TP1 PP4 CP1 EP64 VPP 0 6 MBS3 GBS3072
Appendix
Table 20: Parallelism and training configuration details for benchmark entries reported in Table 11 .
Sparse Mixture-of-Experts (MoE) models offer a powerful way to scale model size without increasing compute, as per-token FLOPs depend only on k active experts rather than the total pool of E experts. Yet, this asymmetry creates an MoE efficiency paradox in practice: adding more experts balloons memory and communication costs, making actual training inefficient. We argue that this bottleneck arises in part because current MoE training allocates too many experts from the beginning, even though early-stage data may not fully utilize such capacity. Motivated by this, we propose EMO, a simple progressive training framework that treats MoE capacity as expandable memory and grows the expert pool over the course of training. EMO explicitly models sparsity in scaling law to derive stage-wise compute-optimal token budgets for progressive expansion. Empirical results show that EMO matches the performance of a fixed-expert setup in large-scale experiments while improving wall-clock efficiency. It offers a surprisingly simple yet effective path to scalable MoE training, preserving the benefits of large expert pools while reducing both training time and GPU cost.
This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models. It is a training paradigm that combines and specializes various existing and novel parallelism techniques at different layers and stages of the Mixture-of-Experts (MoE) model training pipeline. It leverages these techniques to achieve maximal efficiency given the physical constraints of CPU, CPU memory, GPU HBM memory, and the CPU-GPU, GPU-GPU, and node-node communication bandwidth of the GPU cluster. It also contains a novel strategy for the optimizer step to achieve high throughput and memory efficiency, enabling practitioners to conduct lossless pre-training/fine-tuning of trillion-parameter scale models, at a million context length, with just under 12 8x H200 GPU nodes, with state-of-the-art throughput and memory efficiency. In our experiments, MoP delivers 4.7x--8.2x higher per-GPU throughput than a strongly-tuned FSDP2 baseline (with the gap widening at larger scale) and sustains training at context lengths up to 1M tokens, where the baseline runs out of memory beyond 64--128K.
When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. In this work, we study communication-efficient MoE models (CE-MoE), in which we adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth. Compared to conventional models which interleave MoE layers after each token-mixing layer (e.g., attention, Mamba-2), CE-MoE models concentrate expert capacity in a select few routed MoE layers, while maintaining depth by adding additional token-mixing and dense-FFN layers. Across a scaling ladder from 2B to 31.5B total parameters, under matched total and activated parameters, CE-MoE models consistently reduce training cost while matching validation loss and downstream benchmarks with full-MoE baselines. At the 31.5B scale, CE-MoE uses 33.3% fewer GPU-hours while improving average downstream score and inference throughput.