cs.DCOct 4, 2026

Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs

Authors: Javad Mirzaei, Jeebak Mitra

Organizations: Office of CTO Dell Technologies Inc. Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Language Model (LLM) inference has become the dominant workload in modern AI systems, requiring serving infrastructures to maximize throughput while meeting strict latency Service-Level Objectives (SLOs). Since state-of-the-art LLMs exceed the compute and memory capacity of a single GPU, inference is commonly distributed across multiple GPUs using tensor parallelism (TP), pipeline parallelism (PP), or hybrid parallelism (HB). However, selecting the most effective parallelism strategy remains challenging due to complex interactions among computation, communication, pipeline utilization, sequence length, batch size, and model architecture. Existing approaches largely rely on empirical evaluation and provide limited analytical insight into the trade-offs among these strategies, particularly across the distinct prefill and decoding phases of inference. In this paper, we present a unified analytical framework for modeling distributed LLM inference under TP, PP, and HB. The framework decomposes end-to-end latency into computation, inter-GPU communication, and pipeline bubble overhead, and derives analytical models that capture TP collective communication, PP point-to-point communication, and pipeline utilization as functions of hardware, model, and workload characteristics. The model further characterizes the differing execution behavior of prefill and decoding, explaining why PP-oriented configurations favor compute-intensive prefill while TP-oriented configurations reduce decoding latency by eliminating pipeline bubbles. Experiments with modern LLMs on multi-GPU platforms validate the model and confirm the fundamental compute-communication trade-off across parallelism strategies. The framework provides practical guidance for parallelism selection, capacity planning, and optimization of future LLM serving systems.

Figures & tables

Figure 1: Tensor Parallelism Data Flow.
Figure 2: Pipeline Parallelism Data Flow.
CategoryKernel Name
GEMMampere_bf16_s16816gemm_bf16_256x128_ldg8_relu_f2f_stages_64x3_tn
ampere_bf16_s16816gemm_bf16_64x64_ldg8_relu_f2f_stages_64x5_tn
ampere_bf16_s16816gemm_bf16_256x128_ldg8_f2f_stages_64x3_tn
ampere_bf16_s16816gemm_bf16_128x64_ldg8_f2f_stages_64x4_tn
ampere_bf16_s16816gemm_bf16_64x64_sliced1x2_ldg8_f2f_stages_64x6_tn
ampere_bf16_s16816gemm_bf16_64x64_sliced1x2_ldg8_relu_f2f_stages_64x5_tn
Table 1: Sample Kernels.
ParameterValue
GPU Platform8 ×\times NVIDIA A100 SXM4 (80 GB)
GPU Memory80 GB HBM2e per GPU
HBM Bandwidth2.0 TB/s per GPU
Peak Compute312 TFLOPS (FP16/BF16 Tensor Core)
InterconnectNVLink 600 GB/s
Inference Frameworkvllm v0.15.1
Table 2: Experimental Parameters.
Figure 3: TTPcommT^{\rm comm}_{\rm TP} per token vs. batch size for different TP degrees. TP communication is dominated by startup latency at small batch sizes and gradually transitions to bandwidth-limited behavior at large batch sizes.
Figure 4: Effective bandwidth vs. batch size for different TP degrees during (a) Prefill and (b) Decode. Bandwidth utilization grows with communication volume and eventually plateaus, revealing the transition from latency-dominated communication to sustainable inter-connect throughput. Decode remains more latency-limited than prefill due to smaller message sizes, resulting in lower effective bandwidth utilization.
Figure 5: Kernel-level breakdown of (a) TTFT and (b) ITL across different TP degrees for Qwen2.5 - 32B - Instruct. There is a fundamental compute-communication trade-off in TP, where computational savings from distributed GEMM are partially offset by increased NCCL synchronization overhead.
Figure 6: Actual and normalized kernel contributions to TTFT and ITL across various TP degrees for Qwen2.5-32B-Instruct and B=64B=64 . Increasing TP degrees reduces both TTFT and ITL by lowering per-GPU computation, while NCCL synchronization consumes an increasing fraction of latency, illustrating the trade-off between compute reduction and communication overhead.
Figure 7: Kernel-level decomposition of (a) ITL and (b) TTFT across different PP degrees for Qwen2.5 - 32B - Instruct. Decode performance is strongly influenced by pipeline bubbles at small batch sizes, whereas prefill workloads maintain high pipeline utilization and are primarily dominated by computation.
Figure 8: Kernel-level breakdown of (a) ITL during decoding and (b) TTFT during prefill for hybrid TP-PP configurations for Qwen 2.52.5 - 3232 B - Instruct using a total of 88 GPUs. Decode performance is strongly influenced by PP bubble overhead, which diminishes as execution becomes more TP-dominant, whereas prefill latency is primarily determined by TP communication overhead and useful computation.
Figure 9: Trade-off between TP and PP for (a) TTFT during prefill and (b) ITL decode for Qwen 2.52.5 - 1414 B-Instruct across 88 GPUs. Prefill performance favors PP due to lower communication overhead, while decode performance favors TP due to reduced pipeline bubbles. Hybrid configurations provide intermediate behavior by balancing TP synchronization costs against PP idle-time overhead.

Explore similar work

CardsList
Jun 14, 2026cs.AR

Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators

As large language models (LLMs) are increasingly deployed in latency- and cost-sensitive settings, inference efficiency has become a central systems challenge. While GPUs dominate current deployments, a growing number of AI accelerators claim advantages for LLM inference, yet it remains unclear under which conditions such accelerators outperform GPUs in practice. Recent inference systems decompose execution into Prefill and Decode phases, which exhibit distinct computational characteristics and latency metrics, commonly captured by time to first token (TTFT) and time per output token (TPOT). This paper presents a phase-aware evaluation of LLM inference performance across GPUs and emerging AI accelerators using a common model, Llama2-7B. By separately measuring Prefill and Decode performance, we reveal that accelerator advantages differ by phase and metric. Our results show that GPUs consistently excel in the compute-intensive Prefill phase, while GroqRack achieves significantly lower TPOT during Decode (batching not currently supported). However, GPUs regain an advantage in Decode throughput as batch size increases. These findings demonstrate that each platform exhibits distinct phase-dependent strengths. We further analyze heterogeneous Prefill/Decode disaggregation across different accelerator platforms, identifying performance gains and the workload and network conditions under which such gains are realized.

Shun Usami, Venkatram Vishwanath, E. Wes Bethel

Department of Computer Science San Francisco State University San Francisco, CA, 94132 · Argonne National Laboratory Lemont, IL, 60439 · Lawrence Berkeley National Laboratory Berkeley, CA, 94720

LLM Inference OptimizationPrefill
AskPDF
Apr 20, 2026cs.CL

DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models

Existing works on large language model (LLM) decomposition mainly focus on improving performance on downstream tasks, but they ignore the poor parallel inference performance when trying to scale up the model size. To mitigate this important performance issue, this paper introduces DeInfer, a high-performance inference system dedicated to parallel inference of decomposed LLMs. It consists of multiple optimizations to maximize performance and be compatible with state-of-the-art optimization techniques. Extensive experiments are carried out to evaluate DeInfer's performance, where the results demonstrate its superiority, suggesting it can greatly facilitate the parallel inference of decomposed LLMs.

You-Liang Huang, Xinhao Huang, Chengxi Liao +1

Boston University · Boston, Massachusetts, USA · HKUST (Guangzhou) +2

LLM Inference OptimizationEfficient InferenceFast Inference
AskPDF
May 30, 2026cs.AI

Threshold-Based Exclusive Batching for LLM Inference

Mixed batching (MB)--interleaving prefill and decode in a single batch--has become the standard scheduling strategy for large language model (LLM) inference due to its efficiency in maximizing compute and memory utilization. However, through controlled experiments, we find that prefill-decode interference inflates MB's per-step marginal cost above that of pure decode. On the high-bandwidth H200 (4.8 TB/s), this occurs only when decode tokens exceed 80% of the batch; however, on the bandwidth-constrained RTX PRO 6000 (1.792 TB/s), this threshold plummets to just 20%. Consequently, the optimal choice between MB and exclusive batching (EB) fundamentally depends on GPU memory bandwidth, model size, and workload composition. We derive a closed-form condition for this EB-MB performance crossover, along with asymptotically optimal phase-switching thresholds and memory-safe batch sizing for EB. Optimized EB achieves up to 41.9% higher throughput on bandwidth-constrained GPUs, while MB retains its advantage on high-bandwidth hardware with larger models. Our hybrid scheduler EB+ applies this condition online to dynamically switch between EB and MB without manual intervention. Under non-stationary traffic with distribution or concurrency shifts, EB+ attains the highest or near-highest throughput in every setting, outperforming MB by up to 36.4%.

Weifang Zhang, Yuzhou Nie, Bowen Pang +2

The Hong Kong Polytechnic University · UC Santa Barbara · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences +1

LLM Inference OptimizationNvidia
AskPDF