cs.LGMar 3, 2026

Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs

Authors: Wuyue Zhang, Chongdong Huang, Chunbo You, Cheng Gu, Fengjuan Wang, Mou Sun

Organizations: Zhejiang Lab, Hangzhou, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Training large-scale Mixture-of-Experts (MoE) models is bottlenecked by activation memory and expert-parallel communication, yet FP4 training remains impractical on Hopper-class GPUs without native MXFP4 or NVFP4 support. In this work, we present a training recipe that enables MXFP4 efficiency for MoE models on Hopper architectures without native 4-bit computation support. A central challenge is to integrate FP4 into an existing BF16/FP8 hybrid training pipeline without incurring costly precision round-trips (e.g., FP4 ↔\leftrightarrow BF16 ↔\leftrightarrow FP8). We address this challenge by introducing direct FP8-to-FP4 quantization and de-quantization, together with scaling-aware FP4 row-wise to column-wise conversion, enabling FP4 activations and expert-parallel communication with minimal overhead. Core MoE computations are executed in FP8, while activations and expert-parallel communication are compressed using MXFP4, achieving substantial memory and bandwidth savings without degrading convergence. At the 671B parameter scale, our method achieves end-to-end training performance comparable to strong FP8 baselines, while reducing peak activation memory by 14.8% (11.8 GB) and improving training throughput by 12.5%, from 1157 to 1302 tokens per GPU per second. These results show that FP4 efficiency can be practically realized for large-scale MoE training through careful software-hardware co-design, even without native FP4 Tensor Core support.

Figures & tables

Figure 1: MXFP4-based MoE training on Hopper GPUs.
Figure 2: Overview of the MXFP4 hybrid-precision training flow.
Recomputation ScopeBaseline (BF16)Baseline (FP8)Ours (MXFP4)
 TGSMemTGSMemTGSMem
Attn + LN + MoE Experts112268.29%115774.20%115659.40%
Attn + LN + MLPOOM-OOM-124860.62%
MLA Up Proj OnlyOOM-OOM-130270.11%
Table 1: Performance on 671B MoE Model.
Recomputation ScopeBaseline (BF16)Baseline (FP8)Ours (MXFP4)
 TGSMemTGSMemTGSMem
Attn + LN + MoE Experts1725.1653.22%1621.9656.81%1608.6849.96%
Attn + LN + MLP1838.5660.80%1746.2057.03%1616.9049.85%
MLP Up Proj OnlyOOM-1831.2066.13%1722.9858.62%
Table 2: Performance on 236B MoE Model.
Figure 3: Fusion Efficiency (Fused Kernel vs. Dequant + TE Quant). Our fused approach achieves consistent speedups (1.6x–1.9x) by eliminating intermediate memory writes.
Figure 4: Total Quantization Overhead (Ours Full Pipeline vs. TE Baseline). While standard linear layers incur overhead ( << 1.0x), specialized MoE kernels achieve net acceleration ( >> 1.5x).
Figure 5: LM Loss Comparison.
More in this topic

Low-Precision Training

See all 23 papers

Explore similar work

CardsList
May 6, 2026

Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism

Frontier models increasingly adopt Mixture-of-Experts (MoE) architectures to achieve large-model performance at reduced cost. However, training MoE models on HPC platforms is hindered by large memory footprints, frequent large-scale communication across heterogeneous networks, and severe workload imbalance. To characterize these challenges, we develop a mathematical model that quantifies memory, compute, and communication requirements for MoE configurations under various parallelization schemes, verified through micro-benchmarking, code instrumentation, and hardware profiling. Our analysis identifies performance bottlenecks: all-to-all latency at scale from expert parallelism, insufficient compute-communication overlap, low GPU utilization from imbalanced skinny GEMMs, and the absence of platform-aware hybrid parallelization strategies. To address these, we introduce Piper, a framework that leverages resource modeling to identify efficient training strategies for MoE models on target HPC platforms, applying pipeline parallelism with optimized schedules. Piper achieves 2-3.5X higher MFU than state-of-the-art frameworks such as X-MoE, and a novel all-to-all algorithm delivers 1.2-9X bandwidth over vendor implementation.

Sajal Dash, Feiyi Wang

Oak Ridge National Laboratory Oak Ridge, USA

Communication-Efficient Distributed TrainingHigh-Performance ComputingMixture-of-Experts ModelsPipeline Parallelism
AskPDF
Jul 2, 2026

Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models

This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models. It is a training paradigm that combines and specializes various existing and novel parallelism techniques at different layers and stages of the Mixture-of-Experts (MoE) model training pipeline. It leverages these techniques to achieve maximal efficiency given the physical constraints of CPU, CPU memory, GPU HBM memory, and the CPU-GPU, GPU-GPU, and node-node communication bandwidth of the GPU cluster. It also contains a novel strategy for the optimizer step to achieve high throughput and memory efficiency, enabling practitioners to conduct lossless pre-training/fine-tuning of trillion-parameter scale models, at a million context length, with just under 12 8x H200 GPU nodes, with state-of-the-art throughput and memory efficiency. In our experiments, MoP delivers 4.7x--8.2x higher per-GPU throughput than a strongly-tuned FSDP2 baseline (with the gap widening at larger scale) and sustains training at context lengths up to 1M tokens, where the baseline runs out of memory beyond 64--128K.

Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao +3

Salesforce AI Research

Communication-Efficient Distributed TrainingHigh-Performance ComputingEfficient Language Model TrainingMixture-of-Experts ModelsMemory-Efficient Training
AskPDF
Mar 8, 2026

Scalable Training of Mixture-of-Experts Models with Megatron Core

Scaling Mixture-of-Experts (MoE) training introduces systems challenges absent in dense models. Because each token activates only a subset of experts, this sparsity allows total parameters to grow much faster than per-token computation, creating coupled constraints across memory, communication, and computation. Optimizing one dimension often shifts pressure to another, demanding co-design across the full system stack. We address these challenges for MoE training through integrated optimizations spanning memory (fine-grained recomputation, offloading, etc.), communication (optimized dispatchers, overlapping, etc.), and computation (Grouped GEMM, fusions, CUDA Graphs, etc.). The framework also provides Parallel Folding for flexible multi-dimensional parallelism, low-precision training support for FP8 and NVFP4, and efficient long-context training. On NVIDIA GB300 and GB200, it achieves 1,233/1,048 TFLOPS/GPU for DeepSeek-V3-685B and 974/919 TFLOPS/GPU for Qwen3-235B. As a performant, scalable, and production-ready open-source solution, it has been used across academia and industry for training MoE models ranging from billions to trillions of parameters on clusters scaling up to thousands of GPUs. This report explains how these techniques work, their trade-offs, and their interactions at the systems level, providing practical guidance for scaling MoE models with Megatron Core.

Zijie Yan, Hongxiao Bai, Xin Yao +42

NVIDIA

Low-Precision TrainingMixture-of-Experts ModelsCommunication-Efficient Distributed TrainingMemory-Efficient Training
AskPDF