Organizations: Department of Mathematics, National University of Singapore · State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
Tensor parallelism ends every layer with an all-reduce that sits between two dependent matrix multiplications, so its cost cannot be hidden and grows as compute outpaces interconnect bandwidth. Sending 4-bit instead of 16-bit values would cut this cost by nearly 4x, but the all-reduce must add the values it carries, and rounding the sum injects an error at every call. Model quality depends on how these errors accumulate over layers, yet existing methods treat each call on its own. We present Echo, a 4-bit all-reduce for training and inference that controls the accumulated error. Echo moves data in 4 bits and sums in high precision, so each value is quantized at most twice per call for any number of GPUs. The summation reveals each injected error, and Echo feeds it back into later calls, so old errors cancel instead of accumulating. We prove that the accumulated error stays bounded independently of depth. On A100 GPUs, Echo is 2.09x faster than the 16-bit NCCL all-reduce on 1 GB messages. It speeds up the time to first token by up to 1.16x and the training step by up to 1.09x on an idle interconnect, and by up to 1.44x and 1.38x as background traffic grows. It keeps the average accuracy over seven benchmarks within 0.4 points of 16-bit inference on Qwen3-8B. In 10B-token pretraining, its final loss is less than 0.9% above that of 16-bit training. Echo needs no FP8 or FP4 hardware.
Figures & tables
Figure 1: Low-bit tensor-parallel communication as a noise process. A pre-normalization transformer adds each attention and MLP block’s output to the hidden state. In tensor-parallel form the block ends in an all-reduce, and replacing it with the quantized all-reduce leaves the block output in place while adding an error δk at that site. A forward pass has N such sites, and the backward pass injects at the same sites (Sec. 5.4 ).
Figure 2: The quantized P2P all-reduce. (a) For P=4 , each GPU sends only its non-owned segments as 4-bit codes to their owners. Each owner dequantizes the received codes and sums them with its own unquantized segment in high precision. The reduced segment is quantized once for the all-gather, and every GPU, including the owner, dequantizes the same code to obtain identical outputs. Each element is quantized at most twice, independently of P . (b) Tiles are staggered across push, reduce, gather, and tail. Peer-visible flags signal cross-GPU readiness and CUDA events order work within a GPU, allowing quantization and transfers of different tiles to overlap.
Model
Method
PIQA
ARC-e
ARC-c
HellaSwag
WinoGrande
MMLU
MMLU-Pro
Llama-3-8B
16-bit
80.52
77.69
53.39
79.26
73.80
61.54
32.34
4-bit, no compensation
78.18
72.22
48.38
77.35
70.64
58.82
28.76
Ours, 8-bit
80.68
77.77
53.07
79.38
74.03
61.50
31.27
Ours, 4-bit
79.65
76.68
51.79
78.93
70.72
59.01
30.13
FlashComm 8-bit (quoted)
80.58
77.40
52.47
79.09
73.09
–
–
FlashComm 4-bit (quoted)
80.09
75.84
51.02
78.11
70.48
–
–
Table 6: Downstream accuracy (%). The FlashComm rows are quoted from ( Li et al., 2024 ; Li et al., 2025b ) , set in gray, and are read against the 16-bit references of those papers. All other rows are measured in our harness at P=4 and are read against the 16-bit row of the same model. Dashes mark results not reported in the cited work.
Figure 3: Training loss of Qwen3-0.6B at P=4 on 10B tokens of Dolma C4, 16-bit at peak LR 3.0×10−4 against our 4-bit all-reduce at peak LR 3.5×10−4 . Left: full run (50-step moving average, y axis clipped, gray band marks the steps on the right). Right: last 1,500 steps (100-step average, ± 1 s.d. band).
Method
Max LR ( ×10−4 )
Step 1000
Tail-100
Δ
16-bit
3.0
4.095
3.197
–
Ours, 4-bit
3.5
4.042
3.225
+0.027
Table 7: Training loss of Qwen3-0.6B at P=4 on Dolma C4. (a) Full 10B-token training runs. (b) A separate 4-bit learning-rate ablation with batch size 32 and 4,096 steps. Tail-100 is the mean loss over the last 100 steps. In (a), Δ is relative to the 16-bit run, and in (b), Δ3.5 is relative to the run at LR 3.5×10−4 . All differences use unrounded tail-100 losses.
4-bit
8-bit
WT ↓
C4 ↓
WT ↓
C4 ↓
Full method
6.362
9.484
6.070
8.970
Quantizer
(full: zero-aware asymmetric, g=128 )
naive zero-point
7.296
11.260
6.072
8.973
symmetric grid
6.568
9.826
6.075
8.979
Table 8: Component ablations , Llama-3-8B at P=4 , WikiText and C4 perplexity (WT = WikiText), lower is better. Each row changes one component of the method, and the g=256 rows also change the group size. Bold marks the minimum within each bit-width column, excluding the 16-bit reference.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Worked example at P=4 . The quantizer is Eqn. ( 8 ) with b=4 and a group equal to the two values of one segment. Top left: each GPU quantizes the three segments it does not own and keeps its own segment in full precision. Top right: the owner adds the four copies in high precision, three of them dequantized from b -bit codes and one its own. Bottom: the all-gather sends one quantized copy of each segment sum, so every GPU holds the same values. The last row is the error this all-reduce injects, the δk of Fig. 1 . Inputs are shown to one decimal and sums to two.
Handling communication overhead in large-scale tensor-parallel training remains a critical challenge due to the dense, near-zero distributions of intermediate tensors, which exacerbate errors under frequent communication and introduce significant computational overhead during compression. To this end, we propose TACO (Tensor-parallel Adaptive COmmunication compression), a robust FP8-based framework for compressing TP intermediate tensors. First, we employ a data-driven reshaping strategy combined with an Adaptive Scale-Hadamard Transform to enable high-fidelity FP8 quantization, while its Dual-Scale Quantization mechanism ensures numerical stability throughout training. Second, we design a highly fused compression operator to reduce memory traffic and kernel launch overhead, allowing efficient overlap with communication. Finally, we integrate TACO with existing state-of-the-art methods for Data and Pipeline Parallelism to develop a compression-enabled 3D-parallel training framework. Detailed experiments on GPT models and Qwen model demonstrate up to 1.87X end-to-end throughput improvement while maintaining near-lossless accuracy, validating the effectiveness and efficiency of TACO in large-scale training.
Man Liu, Xingchen Liu, Xingjian Tian +8
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · University of Science and Technology of China
Multi-hop all-reduce is the de facto backbone of large model training. As the training scale increases, the network often becomes a bottleneck, motivating the reduction of the volume of transmitted data. Accordingly, recent systems have demonstrated significant acceleration of the training process using gradient quantization. However, these systems are not optimized for multi-hop aggregation, where entries are partially summed multiple times along their aggregation topology. We present DynamiQ, a quantization framework that bridges the gap between quantization best practices and multi-hop aggregation. DynamiQ introduces novel techniques to better represent partial sums, codesigned with a decompress accumulate recompress fused kernel to facilitate fast execution. We extend PyTorch DDP to support DynamiQ over NCCL P2P, and across different LLMs, tasks, and scales, we demonstrate consistent improvement of up to 34.2% over the best among state-of-the-art methods such as Omni-Reduce, THC, and emerging standards such as MXFP4, MXFP6, and MXFP8. Further, DynamiQ is the only evaluated method that consistently reaches near-baseline accuracy (e.g., 99.9% of the BF16 baseline) and does so while significantly accelerating the training.
Wenchen Han, Shay Vargaftik, Michael Mitzenmacher +1
University College London · VMware Research by Broadcom · Harvard University +1
Training large language models is generally done on clusters containing thousands of accelerators, communicating over a high-bandwidth interconnect. Scaling up these clusters is expensive and can become impractical, imposing limits on the size of models that can be trained. Several recent studies have proposed training methods that are less communication intensive, avoiding the need for compute clusters with extremely high interconnect speeds. These low communication training methods still employ a global synchronization step for model parameters, which can be too costly with a high number of participants, as the communication cost scales quadratically with group size. In this work, we propose a novel optimization method, NoLoCo, that does not explicitly synchronize all model parameters during training and does not require any collective communication. NoLoCo implicitly synchronizes model weights via a novel variant of the Nesterov momentum optimizer by partially averaging model weights within randomly selected subgroups. We provide both a theoretical convergence analysis of our optimizer and empirical results from language model training. Our method requires significantly less communication than fully sharded data parallel training and DiLoCo, a widely used low-communication baseline. Moreover, our method avoids global blocking communication, thereby reducing accelerator idle time. Our experiments show that NoLoCo is more communication-efficient than DiLoCo, improving final perplexity by up to 4% and converging up to 4× faster in wall-clock time across a range of worker counts, model sizes, and communication bandwidths.