DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72

Wanqian Li; Jintao Peng; Zongfei Jing; Tianyu Zhang; Ze Long; Xianjie Qiao; Xiaoming Chen; Dongxu Yang

cs.DCApr 2, 2026

DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72

Authors: Wanqian Li, Jintao Peng, Zongfei Jing, Tianyu Zhang, Ze Long, Xianjie Qiao, Xiaoming Chen, Dongxu Yang,

Organizations: NVIDIA

Abstract

Large language model (LLM) inference increasingly depends on multi-GPU execution, yet existing inference parallelization strategies require layer-wise inter-rank synchronization, making end-to-end performance sensitive to workload imbalance. We present DWDP (Distributed Weight Data Parallelism), an inference parallelization strategy that preserves data-parallel execution while offloading MoE weights across peer GPUs and fetching missing experts on demand. By removing collective inter-rank synchronization, DWDP allows each GPU to progress independently. We further address the practical overheads of this design with two optimizations for split-weight management and asynchronous remote-weight prefetch. Implemented in TensorRT-LLM and evaluated with DeepSeek-R1 on GB200 NVL72, DWDP improves end-to-end output TPS/GPU by 8.8% at comparable TPS/user in the 20-100 TPS/user serving range under 8K input sequence length and 1K output sequence length.

Figures & tables

Panel of Figure 1 — Figure 1: Synchronization overhead caused by workload imbalance in DEP. (a) Illustration of how request-level and weight-level imbalance are translated into waiting time in DEP. (b) Kernel breakdown quantifying the synchronization overhead caused by imbalance under DEP. Configuration: DeepSeek-R1 on GB200 with input sequence length/output sequence length (ISL/OSL) = 8K/1 and input ratio 0.8, meaning that the input lengths range from 0.8 $\times$ 8K to 8K.

Figure 2: Overview of DWDP with DWDP group size 4.

Panel of Figure 3 — Figure 3: Roofline-based preliminary analysis for the DeepSeek-R1 context phase on GB200, comparing DWDP4 against DEP4 at batch size 1. The two subplots separately show the compute-to-prefetch ratio and the DEP-to-DWDP runtime ratio. The dashed line at $y=1$ marks the boundary where prefetch can be fully hidden and where DWDP begins to outperform DEP.

Category	DEP4 ( $\mu$ s)	DWDP4 ( $\mu$ s)	$\Delta/T_{\mathrm{DEP4}}$
Attention	269.67	320.56	-3.86%
GroupedGEMM	342.40	337.42	0.38%
DenseGEMM	177.50	189.28	-0.89%
Others	241.69	284.32	-3.23%
Communication	126.74	0.00	9.60%
D2D Copy	0.00	34.00	-2.58%

Table 1: Context-only iteration-latency breakdown of DEP4 and DWDP4 for DeepSeek-R1 under input sequence length

=8

K, ratio

=0.8

, and context-phase maximum number of tokens

=32768

on GB200. The last column reports per-category deltas normalized to the DEP4 iteration latency.

Figure 4: Nsight Systems trace showing many-to-one source-side communication contention in DWDP under max_num_tokens $=16384$ and input sequence lengths ranging from 4K to 8K. Multiple destination ranks concurrently pull remote weights from the same source rank, so the source-side copy engine serializes these requests and exposes compute bubbles.

Config	$C=1$	$C=2$	$C=3$	$C=4$	$C=5$	$C=6$	$C=7$	$C=8$	$C=9$	$C=10$	$C=11$	$C=12$	$C=13$	$C=14$	$C=15$
DWDP3	50.00	50.00	–	–	–	–	–	–	–	–	–	–	–	–	–
DWDP4	44.44	44.44	11.11	–	–	–	–	–	–	–	–	–	–	–	–
DWDP6	40.96	40.96	15.36	2.56	0.16	–	–	–	–	–	–	–	–	–	–
DWDP8	39.66	39.66	16.52	3.67	0.46	0.03	0.00085	–	–	–	–	–	–	–	–
DWDP12	38.55	38.55	17.35	4.63	0.81	0.097	0.0081	0.00046	0.000017	$3.86\times 10^{-7}$	$3.86\times 10^{-9}$	–	–	–	–
DWDP16	38.06	38.06	17.67	5.05	0.99	0.14	0.015	0.0012	0.000077	$3.69\times 10^{-6}$	$1.32\times 10^{-7}$	$3.42\times 10^{-9}$	$6.11\times 10^{-11}$	$6.71\times 10^{-13}$	$3.43\times 10^{-15}$

Table 2: Contention probability

\Pr[C=c]

under the random asynchronous model. Entries are percentages.

Table 3: Ablation study of context-only performance under different workload and configuration settings.

ISL Ratio	MNT	DEP	DWDP + Merge Elim.	Full DWDP
0.5	16384	1.000	0.995	1.081
0.5	32768	1.000	1.140	1.139
0.8	16384	1.000	1.039	1.053
0.8	32768	1.000	1.098	1.109

Table 4: Context-only TPS/GPU under the ISL=8K workload using 1MB slices, normalized to DEP within each row. The last two columns compare DWDP with and without contention mitigation after split-weight merge elimination.

Figure 5: End-to-end Pareto frontier comparison between baseline and DWDP .

TPS/user Range	Avg. DWDP TPS/user Speedup	Avg. DWDP TPS/GPU Speedup
20–30	1.15	1.10
40–50	1.16	1.08
60–70	1.00	1.10
80–90	1.00	1.06
170–180	1.00	0.97

Table 5: End-to-end performance summary of DWDP across target TPS/user ranges.

TPS/user Range	TPS/GPU Speedup	Baseline TTFT (ms)	DWDP TTFT (ms)
20–30	1.10	2538	8314
40–50	1.08	1919	7012
60–70	1.12	965	1640
80–90	1.06	1669	2280
170–180	0.97	494	660

Table 6: Median TTFT comparison across target TPS/user ranges.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Copy-engine-driven remote weight pull across two GPUs. The transfer crosses NVLink and uses the NoC, L2 cache, and DRAM on both GPUs. Interference arises from shared NoC routing, L2 resources, DRAM bandwidth, and power budget.

Figure 7: Illustration of the three communication patterns used in our overlap study. Intermittent Compute inserts large idle gaps between DeepSeek R1 attention modules to maximize power headroom, Long-Duration Overlap uses the longest communication windows while still preserving gaps between compute modules, and Short-Duration Overlap mimics the real DWDP workload with tightly scheduled compute and smaller communication tasks.

Category	Metric	Intermittent Compute	Long-Duration Overlap	Short-Duration Overlap
Power & Thermal	Normalized Kernel Time	1.000	1.049	1.226
Power & Thermal	Normalized GPU Frequency	1.000	0.963	0.798
Memory	max Utilization	50%	45%	47%
Memory	Derived Memory Access Count (GB)	1.7	8.0	6.2
NVLink	Derived NVLink Memory Footprint (GB)	–	6.2	4.7

Appendix

Table 7: GPU metrics for the DeepSeek R1 attention module under three communication-overlap patterns. Kernel time and GPU frequency are normalized to the Intermittent Compute baseline.

Figure 8: Normalized attention-kernel runtime and GPU frequency across the three overlap patterns. The two curves closely follow each other.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
Oct 4, 2026Javad Mirzaei, Jeebak MitraLLM Inference Tensor Parallelism
Office of CTO Dell Technologies Inc. Canada
Unified Static-Dynamic Pruning for Efficient LLM Inference
Jul 24, 2026Jinhyeok Kim, Yejoon Lee, Jaeyoung DoEfficient Neural Network Inference LLM Pruning
Seoul National University Seoul, South Korea
DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models
Apr 20, 2026You-Liang Huang, Xinhao Huang, Chengxi Liao +1LLM Inference Parallel LLM Inference
Boston University · Boston, Massachusetts, USA · HKUST (Guangzhou) +2

Oct 4, 2026J/K move · Enter open · S save

Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs

Javad Mirzaei, Jeebak Mitra

Office of CTO Dell Technologies Inc. Canada

LLM Inference Tensor Parallelism Pipeline Parallelism

Open paper AskPDF arXiv

Large Language Model (LLM) inference has become the dominant workload in modern AI systems, requiring serving infrastructures to maximize throughput while meeting strict latency Service-Level Objectives (SLOs). Since state-of-the-art LLMs exceed the compute and memory capacity of a single GPU, inference is commonly distributed across multiple GPUs using tensor parallelism (TP), pipeline parallelism (PP), or hybrid parallelism (HB). However, selecting the most effective parallelism strategy remains challenging due to complex interactions among computation, communication, pipeline utilization, sequence length, batch size, and model architecture. Existing approaches largely rely on empirical evaluation and provide limited analytical insight into the trade-offs among these strategies, particularly across the distinct prefill and decoding phases of inference. In this paper, we present a unified analytical framework for modeling distributed LLM inference under TP, PP, and HB. The framework decomposes end-to-end latency into computation, inter-GPU communication, and pipeline bubble overhead, and derives analytical models that capture TP collective communication, PP point-to-point communication, and pipeline utilization as functions of hardware, model, and workload characteristics. The model further characterizes the differing execution behavior of prefill and decoding, explaining why PP-oriented configurations favor compute-intensive prefill while TP-oriented configurations reduce decoding latency by eliminating pipeline bubbles. Experiments with modern LLMs on multi-GPU platforms validate the model and confirm the fundamental compute-communication trade-off across parallelism strategies. The framework provides practical guidance for parallelism selection, capacity planning, and optimization of future LLM serving systems.

Abstract

LLM Inference Acceleration

Explore similar work

Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs

Unified Static-Dynamic Pruning for Efficient LLM Inference

DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models