cs.AINov 3, 2026

FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training

Authors: Man Liu, Xingchen Liu, Hairui Zhao, Guangming Tan, Dingwen Tao

Organizations: University of Chinese Academy of Sciences, Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

Abstract

The rapid scaling of large language models (LLMs) has made communication overhead a critical bottleneck in distributed model-parallel training. Although existing low-precision compression methods can effectively reduce communication costs, further reducing the communication footprint to ultra-low-bit formats such as FP4 introduces new challenges, including limited numerical range, sub-byte storage overhead, and increased computational cost from fine-grained quantization. In this paper, we present FP4Quant, a micro-grouped FP4 compression framework for efficient model-parallel LLM training. FP4Quant integrates a micro-grouped dual-scale FP4 quantization mechanism to preserve local tensor distribution characteristics, a warp-cooperative FP4 packing scheme to enable efficient sub-byte data representation, and a hybrid two-level FWHT pipeline to mitigate quantization errors with GPU-aware execution optimization. Experiments on NVIDIA H100 GPUs with Qwen-7B demonstrate up to 1.56×\times end-to-end throughput improvement over BF16, revealing a trade-off between throughput and model quality.

Figures & tables

Explore similar work

CardsList
  1. FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling

    Aug 3, 2026Xianglong Yan, Hong Liu, Chengzhu Bao +4LLM Quantization4-Bit Quantization

  2. SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

    Jun 25, 2026Haoqian Meng, Yilun Luo, Yafei Zhao +44-Bit QuantizationLLM Inference Acceleration

  3. ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

    Jan 12, 2026Haoqian Meng, Yilun Luo, Yafei Zhao +3LLM Quantization4-Bit Quantization