FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training
Organizations: University of Chinese Academy of Sciences, Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
Abstract
The rapid scaling of large language models (LLMs) has made communication overhead a critical bottleneck in distributed model-parallel training. Although existing low-precision compression methods can effectively reduce communication costs, further reducing the communication footprint to ultra-low-bit formats such as FP4 introduces new challenges, including limited numerical range, sub-byte storage overhead, and increased computational cost from fine-grained quantization. In this paper, we present FP4Quant, a micro-grouped FP4 compression framework for efficient model-parallel LLM training. FP4Quant integrates a micro-grouped dual-scale FP4 quantization mechanism to preserve local tensor distribution characteristics, a warp-cooperative FP4 packing scheme to enable efficient sub-byte data representation, and a hybrid two-level FWHT pipeline to mitigate quantization errors with GPU-aware execution optimization. Experiments on NVIDIA H100 GPUs with Qwen-7B demonstrate up to 1.56 end-to-end throughput improvement over BF16, revealing a trade-off between throughput and model quality.
Figures & tables
| Configuration | Method | CMMLU | C-Eval (valid) |
|---|---|---|---|
| BF16 (Ring) | |||
| TP | FP4Quant | ||
| Tahquant | |||
| PP | FP4Quant | ||
| Tahquant | |||
| MP | FP4Quant | ||
| Method | TP | PP | ||||||
|---|---|---|---|---|---|---|---|---|
| Loss | PPL | TFLOP/s | Speedup | Loss | PPL | TFLOP/s | Speedup | |
| BF16 (Ring) | 2.1505 | 8.5893 | 111.0 | 1.00 | 2.1505 | 8.5893 | 189.1 | 1.00 |
| Tahquant | 3.4848 | 32.6163 | 197.4 | 1.78 | 2.1653 | 8.7174 | 189.6 | 1.00 |
| FP4Quant( ) | 2.2306 | 9.3050 | 155.3 | 1.40 | 2.1525 | 8.6105 | 190.1 | 1.01 |
| FP4Quant( ) | 2.4223 | 11.2713 | 171.2 | 1.54 | 2.1552 | 8.6298 | 191.1 | 1.01 |
| FP4Quant( ) | 3.0616 | 21.3613 | 173.3 | 1.56 | 2.1626 | 8.6940 | 190.7 | 1.01 |
Communication-Efficient Distributed Training
Communication-efficient distributed training reduces the data exchanged among devices so models can be trained faster and over limited-bandwidth networks.
See all 245 papers