NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining

Zhida Jiang; Zhaolong Xing; Huichao Chai; Tianxing Sun; Qiang Peng; Baopeng Yuan; Jiaxing Wang; Hua Du

cs.DCApr 8, 2026

NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining

Authors: Zhida Jiang, Zhaolong Xing, Huichao Chai, Tianxing Sun, Qiang Peng, Baopeng Yuan, Jiaxing Wang, Hua Du,

Organizations: Huawei

Abstract

Modern recommendation models have increased to trillions of parameters. As cluster scales expand to O(1k), distributed training bottlenecks shift from computation and memory to data movement, especially lookup and communication latency associated with embeddings. Existing solutions either optimize only one bottleneck or improve throughput by sacrificing training consistency. This paper presents NestPipe, a large-scale decentralized embedding training framework that tackles both bottlenecks while preserving synchronous training semantics. NestPipe exploits two hierarchical sparse parallelism opportunities through nested pipelining. At the inter-batch level, Dual-Buffer Pipelining (DBP) constructs a staleness-free five-stage pipeline through dual-buffer synchronization, mitigating lookup bottlenecks without embedding staleness. At the intra-batch level, we identify the embedding freezing phenomenon, which inspires Frozen-Window Pipelining (FWP) to overlap All2All communication with dense computation via coordinated stream scheduling and key-centric sample clustering. Experiments on production GPU and NPU clusters with 1,536 workers demonstrate that NestPipe achieves up to 3.06x speedup and 94.07% scaling efficiency.

Figures & tables

Fig. 1 : Hybrid decentralized architecture for large-scale recommendation training.

Different Methods	Efficiency		Consistency	Scalability	Orthogonality
Different Methods	Lookup	Communication	Consistency	Scalability	with NestPipe
Asynchronous Training ( e.g. , [ 12 , 13 , 14 , 15 , 36 , 37 , 38 ] )	✓	✗	✗	✗	✗
Embedding Compression ( e.g. , [ 6 , 16 , 27 , 28 , 17 , 29 , 30 , 18 ] )	✓	✓	✗	✗	✓
Embedding Sharding and Scheduling ( e.g. , [ 21 , 22 , 23 , 24 , 25 , 26 ] )	✗	✓	✓	✗	✓
Two-dimensional Sparse Parallelism [ 8 ]	✗	✓	✗	✓	✓
NestPipe (Ours)	✓	✓	✓	✓	-

TABLE I : Comprehensive comparison of existing works with NestPipe.

Panel of Figure 3 — Fig. 2 : Impact of cluster scale on sparse lookup and communication overhead.

Fig. 4 : Dual-buffer synchronization in DBP strategy.

Fig. 5 : Implementation of FWP strategy through coordinated communication and computation stream scheduling.

Cluster	Dataset	Method	Step	Speedup	Ablation Study
Cluster	Dataset	Method	Latency	Speedup	Lookup	Comm.
	Industrial	TorchRec	5793.83	1.00 $\times$	2870.99	1207.85
1536		2D-SP	4914.01	1.18 $\times$	2766.68	438.36
NPUs		UniEmb	2919.76	1.98 $\times$	36.21	1169.01
		NestPipe	1895.98	3.06 $\times$	30.19	154.23
		TorchRec	4207.94	1.00 $\times$	856.63	312.19

TABLE II : Overall latency (ms) of different methods on GPU and NPU clusters, and ablation study of DBP and FWP.

Fig. 6 : The training loss and accuracy curve of different methods.

# Worker	TorchRec			2D-SP			UniEmb			NestPipe
# Worker	QPS	Speedup	Scaling	QPS	Speedup	Scaling	QPS	Speedup	Scaling	QPS	Speedup	Scaling
128	0.26	1.00 $\times$	-	0.27	1.04 $\times$	-	0.33	1.27 $\times$	-	0.37	1.42 $\times$	-
256	0.47	1.00 $\times$	91.23%	0.49	1.04 $\times$	91.36%	0.63	1.34 $\times$	96.13%	0.72	1.53 $\times$	98.39%
512	0.68	1.00 $\times$	66.64%	0.75	1.10 $\times$	69.64%	1.17	1.72 $\times$	89.21%	1.43	2.10 $\times$	97.33%
1024	1.09	1.00 $\times$	53.36%	1.22	1.12 $\times$	56.67%	2.04	1.87 $\times$	78.00%	2.80	2.57 $\times$	95.63%
1536	1.36	1.00 $\times$	44.34%	1.60	1.18 $\times$	49.32%	2.65	1.98 $\times$	67.62%	4.14	3.06 $\times$	94.07%

TABLE III : Scaling performance comparison of different methods. QPS is reported in the scale of

10^{5}

. The speedup is computed against TorchRec under the same cluster configuration. The scaling factor is normalized to the 128-worker baseline.

Fig. 7 : Step latency breakdown of different methods for varying cluster sizes.

Fig. 8 : Resource utilization ratio of different methods for varying cluster sizes.

Panel of Figure 12 — Fig. 9 : Impact of micro-batch size on step latency and exposed comm. ratio under a constant batch size of 512.

Fig. 10 : Step latency breakdown for varying embedding dimensions, dense layers, and sequence lengths.

Method	Total Comm. Latency (ms)	Exposed Comm. Latency (ms)	QPS	Scaling
TorchRec	1207.85	1207.85	1.36	44.34%
2D-SP	438.36	438.36	1.60	49.32%
NestPipe	1185.60	154.23	4.14	94.07%
NestPipe+2D-SP	452.34	55.64	4.32	97.17%

TABLE IV : Integration of NestPipe and 2D-SP optimizations on 1,536 workers. QPS is reported in the scale of

10^{5}

, and the scaling factor is normalized to the 128-worker baseline.

Distributed Training

Distributed training coordinates computation across multiple devices or machines to train models that exceed single-device limits.

See all 129 papers

FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost
Apr 27, 2026Chenhao Feng, Haoli Zhang, Shakhzod Ali-Zade +17Sequential Recommendation Communication-Efficient Distributed Training
1Meta Inc.
Piper: A Programmable Distributed Training System
Jun 9, 2026Megan Frisella, Shubham Tiwari, Andy Ruan +5Distributed Optimization Distributed Training
University of Washington Seattle, WA, USA · University of Washington and Shanghai Jiao Tong University Seattle, WA, USA
ERASE: EaRly bAckpropagation SchEdule for Faster Training of Modern Recommendation Systems
Aug 19, 2026Ergan Shang, Flavio Sales TruzziDeep Learning Optimization Backpropagation
Carnegie Mellon University · Meta Inc.

Apr 27, 2026J/K move · Enter open · S save

FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost

Chenhao Feng, Haoli Zhang, Shakhzod Ali-Zade, Yanli Zhao+16

1Meta Inc.

Sequential Recommendation Communication-Efficient Distributed Training Distributed Optimization Distributed Training

Open paper AskPDF arXiv

Modern industrial Deep Learning Recommendation Models typically extract user preferences through the analysis of sequential interaction histories, subsequently generating predictions based on these derived interests. The inherent heterogeneity in data characteristics frequently result in substantial under-utilization of computational resources during large-scale training, primarily due to computational bubbles caused by severe stragglers and slow blocking communications. This paper introduces FreeScale, a solution designed to (1) mitigate the straggler problem through meticulously load balanced input samples (2) minimize the blocking communication by overlapping prioritized embedding communications with computations (3) resolve the GPU resource competition during computation and communication overlapping by communicating through SM-Free techniques. Empirical evaluation demonstrates that FreeScale achieves up to 90.3% reduction in computational bubbles when applied to real-world workloads running on 256 H100 GPUs.

Abstract

Distributed Training

Explore similar work

FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost

Piper: A Programmable Distributed Training System

ERASE: EaRly bAckpropagation SchEdule for Faster Training of Modern Recommendation Systems