Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference

Xingyu Xie; Xiaoyong Xie; Zhouchen Lin; Kim-Chuan Toh

cs.AIOct 9, 2026

Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference

Authors: Xingyu Xie, Xiaoyong Xie, Zhouchen Lin, Kim-Chuan Toh

Organizations: Department of Mathematics, National University of Singapore · State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University

Abstract

Tensor parallelism ends every layer with an all-reduce that sits between two dependent matrix multiplications, so its cost cannot be hidden and grows as compute outpaces interconnect bandwidth. Sending 4-bit instead of 16-bit values would cut this cost by nearly 4x, but the all-reduce must add the values it carries, and rounding the sum injects an error at every call. Model quality depends on how these errors accumulate over layers, yet existing methods treat each call on its own. We present Echo, a 4-bit all-reduce for training and inference that controls the accumulated error. Echo moves data in 4 bits and sums in high precision, so each value is quantized at most twice per call for any number of GPUs. The summation reveals each injected error, and Echo feeds it back into later calls, so old errors cancel instead of accumulating. We prove that the accumulated error stays bounded independently of depth. On A100 GPUs, Echo is 2.09x faster than the 16-bit NCCL all-reduce on 1 GB messages. It speeds up the time to first token by up to 1.16x and the training step by up to 1.09x on an idle interconnect, and by up to 1.44x and 1.38x as background traffic grows. It keeps the average accuracy over seven benchmarks within 0.4 points of 16-bit inference on Qwen3-8B. In 10B-token pretraining, its final loss is less than 0.9% above that of 16-bit training. Echo needs no FP8 or FP4 hardware.

Figures & tables

Panel of Figure 1 — Figure 1: Low-bit tensor-parallel communication as a noise process. A pre-normalization transformer adds each attention and MLP block’s output to the hidden state. In tensor-parallel form the block ends in an all-reduce, and replacing it with the quantized all-reduce leaves the block output in place while adding an error $\bm{\delta}_{k}$ at that site. A forward pass has $N$ such sites, and the backward pass injects at the same sites (Sec. 5.4 ).

Panel of Figure 2 — Figure 2: The quantized P2P all-reduce. (a) For $P=4$ , each GPU sends only its non-owned segments as 4-bit codes to their owners. Each owner dequantizes the received codes and sums them with its own unquantized segment in high precision. The reduced segment is quantized once for the all-gather, and every GPU, including the owner, dequantizes the same code to obtain identical outputs. Each element is quantized at most twice, independently of $P$ . (b) Tiles are staggered across push, reduce, gather, and tail. Peer-visible flags signal cross-GPU readiness and CUDA events order work within a GPU, allowing quantization and transfers of different tiles to overlap.

Model	Method	PIQA	ARC-e	ARC-c	HellaSwag	WinoGrande	MMLU	MMLU-Pro
Llama-3-8B	16-bit	80.52	77.69	53.39	79.26	73.80	61.54	32.34
	4-bit, no compensation	78.18	72.22	48.38	77.35	70.64	58.82	28.76
	Ours, 8-bit	80.68	77.77	53.07	79.38	74.03	61.50	31.27
	Ours, 4-bit	79.65	76.68	51.79	78.93	70.72	59.01	30.13
	FlashComm 8-bit (quoted)	80.58	77.40	52.47	79.09	73.09	–	–
	FlashComm 4-bit (quoted)	80.09	75.84	51.02	78.11	70.48	–	–

Table 6: Downstream accuracy (%). The FlashComm rows are quoted from ( Li et al., 2024 ; Li et al., 2025b ) , set in gray, and are read against the 16-bit references of those papers. All other rows are measured in our harness at

P=4

and are read against the 16-bit row of the same model. Dashes mark results not reported in the cited work.

Figure 3: Training loss of Qwen3-0.6B at

P=4

on 10B tokens of Dolma C4, 16-bit at peak LR

3.0\times 10^{-4}

against our 4-bit all-reduce at peak LR

3.5\times 10^{-4}

. Left: full run (50-step moving average, y axis clipped, gray band marks the steps on the right). Right: last 1,500 steps (100-step average,

\pm

1 s.d. band).

Method	Max LR ( $\times 10^{-4}$ )	Step 1000	Tail-100	$\Delta$
16-bit	3.0	4.095	3.197	–
Ours, 4-bit	3.5	4.042	3.225	+0.027

Table 7: Training loss of Qwen3-0.6B at

P=4

on Dolma C4. (a) Full 10B-token training runs. (b) A separate 4-bit learning-rate ablation with batch size 32 and 4,096 steps. Tail-100 is the mean loss over the last 100 steps. In (a),

\Delta

is relative to the 16-bit run, and in (b),

\Delta_{3.5}

is relative to the run at LR

3.5\times 10^{-4}

. All differences use unrounded tail-100 losses.

Quantizer
	4-bit		8-bit
	WT $\downarrow$	C4 $\downarrow$	WT $\downarrow$	C4 $\downarrow$
Full method	6.362	9.484	6.070	8.970
(full: zero-aware asymmetric, $g=128$ )
naive zero-point	7.296	11.260	6.072	8.973
symmetric grid	6.568	9.826	6.075	8.979

Table 8: Component ablations , Llama-3-8B at

P=4

, WikiText and C4 perplexity (WT = WikiText), lower is better. Each row changes one component of the method, and the

g=256

rows also change the group size. Bold marks the minimum within each bit-width column, excluding the 16-bit reference.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Worked example at $P=4$ . The quantizer is Eqn. ( 8 ) with $b=4$ and a group equal to the two values of one segment. Top left: each GPU quantizes the three segments it does not own and keeps its own segment in full precision. Top right: the owner adds the four copies in high precision, three of them dequantized from $b$ -bit codes and one its own. Bottom: the all-gather sends one quantized copy of each segment sum, so every GPU holds the same values. The last row is the error this all-reduce injects, the $\bm{\delta}_{k}$ of Fig. 1 . Inputs are shown to one decimal and sums to two.

Error Feedback

See all 22 papers

TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training
Apr 27, 2026Man Liu, Xingchen Liu, Xingjian Tian +8Communication-Efficient Distributed Training Tensor Parallelism
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · University of Science and Technology of China
DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce
Feb 9, 2026Wenchen Han, Shay Vargaftik, Michael Mitzenmacher +1Communication-Efficient Distributed Training High-Performance Computing
University College London · VMware Research by Broadcom · Harvard University +1
NoLoCo: No-all-reduce Low Communication Training Method for Large Models
Jun 12, 2025Jari Kolehmainen, Nikolay Blagoev, Semih Kara +3Communication-Efficient Distributed Training Decentralized Learning
1Gensyn

Apr 27, 2026J/K move · Enter open · S save

TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training

Man Liu, Xingchen Liu, Xingjian Tian, Bing Lu+7

Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · University of Science and Technology of China

Communication-Efficient Distributed Training Tensor Parallelism Distributed Training

Open paper AskPDF arXiv

Handling communication overhead in large-scale tensor-parallel training remains a critical challenge due to the dense, near-zero distributions of intermediate tensors, which exacerbate errors under frequent communication and introduce significant computational overhead during compression. To this end, we propose TACO (Tensor-parallel Adaptive COmmunication compression), a robust FP8-based framework for compressing TP intermediate tensors. First, we employ a data-driven reshaping strategy combined with an Adaptive Scale-Hadamard Transform to enable high-fidelity FP8 quantization, while its Dual-Scale Quantization mechanism ensures numerical stability throughout training. Second, we design a highly fused compression operator to reduce memory traffic and kernel launch overhead, allowing efficient overlap with communication. Finally, we integrate TACO with existing state-of-the-art methods for Data and Pipeline Parallelism to develop a compression-enabled 3D-parallel training framework. Detailed experiments on GPT models and Qwen model demonstrate up to 1.87X end-to-end throughput improvement while maintaining near-lossless accuracy, validating the effectiveness and efficiency of TACO in large-scale training.

Abstract

Error Feedback

Explore similar work

TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training

DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce

NoLoCo: No-all-reduce Low Communication Training Method for Large Models