Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

Javad Mirzaei; Jeebak Mitra

cs.DCOct 14, 2026

Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

Authors: Javad Mirzaei, Jeebak Mitra

Organizations: Office of CTO Dell Technologies Inc. Canada

Abstract

Large Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources. This challenge arises because inference consists of two fundamentally different phases: prefill, which determines time-to-first-token (TTFT), and decode, which determines inter-token latency (ITL). The use of tensor parallelism (TP), pipeline parallelism (PP), and hybrid configurations exposes different trade-offs between these latency dimensions, making the optimal parallelism configuration highly dependent on workload characteristics, latency requirements, and context length. Moreover, despite the widespread adoption of distributed inference, there is still limited understanding of how parallelism configurations should be selected under varying workload conditions and service-level objectives (SLO). In this paper, we develop a systematic framework for characterizing LLM inference parallelism via throughput-latency trade-offs, SLO-aware serving, and long-context scalability. We introduce the notion of parallelism operating regions by constructing TTFT-ITL feasible sets and Pareto frontiers across different workload compositions, thereby revealing a systematic transition from TP-dominant to PP-dominant configurations as workloads become increasingly prefill-heavy. We further formulate SLO-aware parallelism selection using goodput, showing that stringent TTFT requirements favor PP-dominated configurations, whereas relaxed latency constraints progressively shift the optimality toward TP-dominated configurations with higher serving capacity. Finally, we demonstrate that increasing context length fundamentally changes the relative advantages of parallelism configurations, with PP-dominant configurations exhibiting more favorable scalability and SLO attainment in the long-context regime.

Figures & tables

$p$	$(8,1)$	$(4,2)$	$(2,4)$	$(1,8)$
Notation	TP8	TP4PP2	TP2PP4	PP8

Table 1: Notation mapping.

Figure 1: TTFT-ITL feasible region and Pareto frontier for different parallelism configurations. TP-dominant configurations achieve lower ITL but higher TTFT, while PP-dominant configurations achieve lower TTFT at the expense of higher ITL. As workloads transition from decode-heavy to prefill-heavy, the Pareto-optimal operating point shifts systematically from TP-dominant configurations, through hybrid configurations, toward PP-dominant configurations.

Figure 2: TTFT vs. ITL for various OSL. The effect of OSL is not uniform across parallelism configurations. Longer OSL mainly improves ITL through better amortization of decode overheads, with the largest benefit observed for TP-dominant configurations.

Figure 3: Left Plot: TTFT distribution (Prefill), Right plot: ITL distribution (decode). PP-dominant configurations achieve lower TTFT, while TP-dominant configurations achieve lower ITL, thereby highlighting the latency trade-off between prefill and decoding across different parallelisms.

Figure 4: Latency compliance and serving capacity trade-off for fixed workload

w=(1200,2800,256)

with

L_{\rm seq}=4000

. Filled marker in each

T_{s}

, indicates the parallelism configuration that yields the maximum goodput for the corresponding

T_{s}

Figure 5: Goodput vs. TTFT SLO for fixed workload

w=(1200,2800,256)

where

L_{\rm seq}=4000

. As TTFT SLO is relaxed, the optimal parallelism shifts from PP-dominant configurations to TP-dominant configurations.

Figure 6: Goodput vs. Sequence Length for workload

w=(L_{\rm in},1000,128)

where

L_{\rm in}

varies to construct values of x-axis. Left plot:

T_{s}=5

(s). Right Plot:

T_{s}=2

(s). Increasing sequence length progressively favors PP-dominant parallelism over TP-dominant parallelism, highlighting PP’s stronger scalability for long context workloads.

Figure 7: CDF vs. Sequence Length for workload

w=(L_{\rm in},1000,128)

where

L_{\rm in}

varies to construct values of x-axis. Left plot:

T_{s}=5

(s). Right plot:

T_{s}=2

(s). PP-dominant configurations maintain higher SLO attainment as sequence length increases, especially under tight TTFT SLOs.

Parallel LLM Inference

See all 16 papers

Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
Oct 4, 2026Javad Mirzaei, Jeebak MitraLLM Inference Tensor Parallelism
Office of CTO Dell Technologies Inc. Canada
Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks
Mar 5, 2026Burak Topcu, Musa Oguzhan Cim, Poovaiah Palangappa +2Parallel LLM Inference LLM Inference
The Pennsylvania State University, State College, Pennsylvania, USA · Advanced Micro Devices, San Jose, California, USA · Advanced Micro Devices, Portland, Oregon, USA
Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Clouds
Apr 8, 2026Jiaming Cheng, Duong Tung NguyenLLM Serving LLM Inference Scheduling
The Ira A. Fulton Schools of Engineering, Arizona State University, Tempe, AZ, USA.

Oct 4, 2026J/K move · Enter open · S save

Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs

Javad Mirzaei, Jeebak Mitra

Office of CTO Dell Technologies Inc. Canada

LLM Inference Tensor Parallelism Pipeline Parallelism

Open paper AskPDF arXiv

Large Language Model (LLM) inference has become the dominant workload in modern AI systems, requiring serving infrastructures to maximize throughput while meeting strict latency Service-Level Objectives (SLOs). Since state-of-the-art LLMs exceed the compute and memory capacity of a single GPU, inference is commonly distributed across multiple GPUs using tensor parallelism (TP), pipeline parallelism (PP), or hybrid parallelism (HB). However, selecting the most effective parallelism strategy remains challenging due to complex interactions among computation, communication, pipeline utilization, sequence length, batch size, and model architecture. Existing approaches largely rely on empirical evaluation and provide limited analytical insight into the trade-offs among these strategies, particularly across the distinct prefill and decoding phases of inference. In this paper, we present a unified analytical framework for modeling distributed LLM inference under TP, PP, and HB. The framework decomposes end-to-end latency into computation, inter-GPU communication, and pipeline bubble overhead, and derives analytical models that capture TP collective communication, PP point-to-point communication, and pipeline utilization as functions of hardware, model, and workload characteristics. The model further characterizes the differing execution behavior of prefill and decoding, explaining why PP-oriented configurations favor compute-intensive prefill while TP-oriented configurations reduce decoding latency by eliminating pipeline bubbles. Experiments with modern LLMs on multi-GPU platforms validate the model and confirm the fundamental compute-communication trade-off across parallelism strategies. The framework provides practical guidance for parallelism selection, capacity planning, and optimization of future LLM serving systems.

Abstract

Parallel LLM Inference

Explore similar work

Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs

Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks

Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Clouds