Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks
Authors: Burak Topcu, Musa Oguzhan Cim, Poovaiah Palangappa, Meena Arunachalam, Mahmut Taylan Kandemir
Organizations: The Pennsylvania State University, State College, Pennsylvania, USA · Advanced Micro Devices, San Jose, California, USA · Advanced Micro Devices, Portland, Oregon, USA
Breakthroughs in the generative AI domain have fueled an explosion of large language model (LLM)-powered applications, whose workloads fundamentally consist of sequences of inferences through transformer architectures. Within this rapidly expanding ecosystem, dense LLMs--those that activate all model parameters for each token generation--form the foundation for advanced expert-based variants. Dense models continue to dominate because of their strong generalization ability, scalability, ease of fine-tuning, and versatility across diverse tasks. In LLM inference systems, performance is mainly characterized by latency, response time, and throughput (i.e., tokens generated per unit of time). Latency and throughput are inherently coupled: optimizing for one often comes at the expense of the other. Moreover, batching strategies and parallelism configurations, which are essential when dense model parameters exceed device memory capacity, can significantly affect both latency and overall system throughput. This paper (i) investigates the workloads of two representative dense LLMs--Llama-3.1-70B and Llama-3.1-405B, focusing in particular on intra-node parallelization schemes, (ii) analyzes how input characteristics, batching, and parallelism strategies influence latency flexibility and the latency-throughput tradeoff, and (iii) identifies key performance bottlenecks that inform design choices for meeting service-level agreements (SLAs) and sustaining inference quality. Our empirical evaluations reveal that Tensor Parallelism (TP) improves the latency objectives while Pipeline Parallelism (PP) is better-suited for throughput-oriented applications. We highlight that their hybrid usage by controlling the TP and PP degrees provides control over the latency-throughput interplay.
Figures & tables
70B Model
405B Model
Number of transformer blocks
80
126
Hidden dimension size
8192
16384
Intermediate dimension size
28672
53248
Number of attention heads
64
128
Attention head dimension
128
128
Total number of parameters
70B
405B
Table 1. Architecture specifications for Llama 3.1-70B/-405B.
Figure 1. Figures (a,c) present simulator transformer pass times in prefill and decode normalized to silicon for Llama 3.1-70B/-405B models. Figures (b) and (d) break down kernel execution times within transformer blocks in prefill and decode for the silicon, and reveal kernel-wise execution comparison errors w.r.t. transformer execution times on the right side y-axis. Simulator validation.
Figure 2. An illustrative overview of the Tensor Parallelism workflow for LLM inference. An illustrative overview of the Tensor Parallelism workflow for LLM inference.
Figure 3. A schematic overview of the Pipeline Parallelism and its workflow distribution for LLM Inference. A schematic overview of the Pipeline Parallelism and its workflow distribution for LLM Inference.
Figure 4. Input and output sequence length distribution for LongAlpaca, MLPerf, and the combination of BSH, GSM8K, and HumanEval datasets on Llama 3.1-70B and -405B models. Input and output sequence length distribution for LongAlpaca, MLPerf, and a combination of BSH, GSM8K, and HumanEval datasets on Llama 3.1-70B and -405B models.
Figure 5. Normalized TTFT and TPOT values when serving inferences for datasets in (a) and (b) through Llama 3.1-70B. Normalized TTFT and TPOT values when serving inferences through Llama 3.1-70B.
Datasets
ISL
OSL
TOTAL
70B
LongAlpaca ( Chen et al., 2024 )
9092
208
9350
BBH, GSM8K, Human Evaluation ( Suzgun et al., 2022 ; Cobbe et al., 2021 ; Chen et al., 2021 )
106
26
132
405B
MLPerf ( MLCommons, 2025b )
9428
684
10112
BBH, GSM8K, Human Evaluation
89
20
110
Table 2. Average input and output sequence lengths of the evaluation datasets when their inferences are served through target LLMs.
GPU
HBM Mem.
Arch.
Precision Support
Inter Connect BW
MI325x ( AMD, 2024 )
256GB
CDNA-3
64, 32, 16, and 8-bit
128GB/s bidirect
MI355x ( AMD, 2025 )
288GB
CDNA-4
64, 32, 16, 8 and 4-bit
153.6GB/s bidirect
Table 3. GPU specifications.
Figure 6. Normalized TTFT and TPOT values for Llama 3.1-405B inference serving with varying parallelization techniques & depths, and batch sizes. Normalized TTFT and TPOT values for Llama 3.1-405B inference serving with varying parallelization techniques & depths, and batch sizes.
Figure 7. Impact of TP size and interconnection link speed on TTFT, all-reduce and P2P communication durations in TP and PP. Impact of interconnection link speed on the TTFT latency in PP and TP.
Figure 8. Normalized output TPS values for inference serving through Llama 3.1-70B (a) and -405B (b) models. Normalized throughput (output TPS) values for inference serving through Llama 3.1-70B (a) and 405B (b) models.