Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs

Javad Mirzaei; Jeebak Mitra

cs.DCOct 4, 2026

Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs

Authors: Javad Mirzaei, Jeebak Mitra

Organizations: Office of CTO Dell Technologies Inc. Canada

Abstract

Large Language Model (LLM) inference has become the dominant workload in modern AI systems, requiring serving infrastructures to maximize throughput while meeting strict latency Service-Level Objectives (SLOs). Since state-of-the-art LLMs exceed the compute and memory capacity of a single GPU, inference is commonly distributed across multiple GPUs using tensor parallelism (TP), pipeline parallelism (PP), or hybrid parallelism (HB). However, selecting the most effective parallelism strategy remains challenging due to complex interactions among computation, communication, pipeline utilization, sequence length, batch size, and model architecture. Existing approaches largely rely on empirical evaluation and provide limited analytical insight into the trade-offs among these strategies, particularly across the distinct prefill and decoding phases of inference. In this paper, we present a unified analytical framework for modeling distributed LLM inference under TP, PP, and HB. The framework decomposes end-to-end latency into computation, inter-GPU communication, and pipeline bubble overhead, and derives analytical models that capture TP collective communication, PP point-to-point communication, and pipeline utilization as functions of hardware, model, and workload characteristics. The model further characterizes the differing execution behavior of prefill and decoding, explaining why PP-oriented configurations favor compute-intensive prefill while TP-oriented configurations reduce decoding latency by eliminating pipeline bubbles. Experiments with modern LLMs on multi-GPU platforms validate the model and confirm the fundamental compute-communication trade-off across parallelism strategies. The framework provides practical guidance for parallelism selection, capacity planning, and optimization of future LLM serving systems.

Figures & tables

Figure 2: Pipeline Parallelism Data Flow.

Category	Kernel Name
GEMM	ampere_bf16_s16816gemm_bf16_256x128_ldg8_relu_f2f_stages_64x3_tn
	ampere_bf16_s16816gemm_bf16_64x64_ldg8_relu_f2f_stages_64x5_tn
	ampere_bf16_s16816gemm_bf16_256x128_ldg8_f2f_stages_64x3_tn
	ampere_bf16_s16816gemm_bf16_128x64_ldg8_f2f_stages_64x4_tn
	ampere_bf16_s16816gemm_bf16_64x64_sliced1x2_ldg8_f2f_stages_64x6_tn
	ampere_bf16_s16816gemm_bf16_64x64_sliced1x2_ldg8_relu_f2f_stages_64x5_tn

Table 1: Sample Kernels.

Parameter	Value
GPU Platform	8 $\times$ NVIDIA A100 SXM4 (80 GB)
GPU Memory	80 GB HBM2e per GPU
HBM Bandwidth	2.0 TB/s per GPU
Peak Compute	312 TFLOPS (FP16/BF16 Tensor Core)
Interconnect	NVLink 600 GB/s
Inference Framework	vllm v0.15.1

Table 2: Experimental Parameters.

Figure 3:

T^{\rm comm}_{\rm TP}

per token vs. batch size for different TP degrees. TP communication is dominated by startup latency at small batch sizes and gradually transitions to bandwidth-limited behavior at large batch sizes.

Panel of Figure 4 — Figure 4: Effective bandwidth vs. batch size for different TP degrees during (a) Prefill and (b) Decode. Bandwidth utilization grows with communication volume and eventually plateaus, revealing the transition from latency-dominated communication to sustainable inter-connect throughput. Decode remains more latency-limited than prefill due to smaller message sizes, resulting in lower effective bandwidth utilization.

Panel of Figure 5 — Figure 5: Kernel-level breakdown of (a) TTFT and (b) ITL across different TP degrees for Qwen2.5 - 32B - Instruct. There is a fundamental compute-communication trade-off in TP, where computational savings from distributed GEMM are partially offset by increased NCCL synchronization overhead.

Panel of Figure 6 — Figure 6: Actual and normalized kernel contributions to TTFT and ITL across various TP degrees for Qwen2.5-32B-Instruct and $B=64$ . Increasing TP degrees reduces both TTFT and ITL by lowering per-GPU computation, while NCCL synchronization consumes an increasing fraction of latency, illustrating the trade-off between compute reduction and communication overhead.

Panel of Figure 7 — Figure 7: Kernel-level decomposition of (a) ITL and (b) TTFT across different PP degrees for Qwen2.5 - 32B - Instruct. Decode performance is strongly influenced by pipeline bubbles at small batch sizes, whereas prefill workloads maintain high pipeline utilization and are primarily dominated by computation.

Panel of Figure 8 — Figure 8: Kernel-level breakdown of (a) ITL during decoding and (b) TTFT during prefill for hybrid TP-PP configurations for Qwen $2.5$ - $32$ B - Instruct using a total of $8$ GPUs. Decode performance is strongly influenced by PP bubble overhead, which diminishes as execution becomes more TP-dominant, whereas prefill latency is primarily determined by TP communication overhead and useful computation.

Panel of Figure 9 — Figure 9: Trade-off between TP and PP for (a) TTFT during prefill and (b) ITL decode for Qwen $2.5$ - $14$ B-Instruct across $8$ GPUs. Prefill performance favors PP due to lower communication overhead, while decode performance favors TP due to reduced pipeline bubbles. Hybrid configurations provide intermediate behavior by balancing TP synchronization costs against PP idle-time overhead.

Explore similar work

Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators
Jun 14, 2026Shun Usami, Venkatram Vishwanath, E. Wes BethelLLM Inference Optimization Prefill
Department of Computer Science San Francisco State University San Francisco, CA, 94132 · Argonne National Laboratory Lemont, IL, 60439 · Lawrence Berkeley National Laboratory Berkeley, CA, 94720
DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models
Apr 20, 2026You-Liang Huang, Xinhao Huang, Chengxi Liao +1LLM Inference Optimization Efficient Inference
Boston University · Boston, Massachusetts, USA · HKUST (Guangzhou) +2
Threshold-Based Exclusive Batching for LLM Inference
May 30, 2026Weifang Zhang, Yuzhou Nie, Bowen Pang +2LLM Inference Optimization Nvidia
The Hong Kong Polytechnic University · UC Santa Barbara · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences +1

Jun 14, 2026, cs.ARJ/K move · Enter open · S save

Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators

Shun Usami, Venkatram Vishwanath, E. Wes Bethel

Department of Computer Science San Francisco State University San Francisco, CA, 94132 · Argonne National Laboratory Lemont, IL, 60439 · Lawrence Berkeley National Laboratory Berkeley, CA, 94720

LLM Inference Optimization Prefill

Open paper AskPDF arXiv

As large language models (LLMs) are increasingly deployed in latency- and cost-sensitive settings, inference efficiency has become a central systems challenge. While GPUs dominate current deployments, a growing number of AI accelerators claim advantages for LLM inference, yet it remains unclear under which conditions such accelerators outperform GPUs in practice. Recent inference systems decompose execution into Prefill and Decode phases, which exhibit distinct computational characteristics and latency metrics, commonly captured by time to first token (TTFT) and time per output token (TPOT). This paper presents a phase-aware evaluation of LLM inference performance across GPUs and emerging AI accelerators using a common model, Llama2-7B. By separately measuring Prefill and Decode performance, we reveal that accelerator advantages differ by phase and metric. Our results show that GPUs consistently excel in the compute-intensive Prefill phase, while GroqRack achieves significantly lower TPOT during Decode (batching not currently supported). However, GPUs regain an advantage in Decode throughput as batch size increases. These findings demonstrate that each platform exhibits distinct phase-dependent strengths. We further analyze heterogeneous Prefill/Decode disaggregation across different accelerator platforms, identifying performance gains and the workload and network conditions under which such gains are realized.

Abstract

Explore similar work

Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators

DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models

Threshold-Based Exclusive Batching for LLM Inference