Pipeline Parallelism

Follow this topic Ask about this topic

Pipeline parallelism assigns consecutive model layers to different devices, then passes activations and gradients between stages so larger models can run across limited memory and compute. Training methods such as GPipe, 1F1B, interleaved schedules, and PipeDream trade pipeline bubbles and activation memory against synchronization and stale weights; inference systems use stage scheduling and micro-batching to balance throughput with latency. Current work adapts partitioning and schedules to heterogeneous hardware, long sequences, and mixture-of-experts models, while also overlapping communication or compressing activations. A central tension is improving device utilization without making communication, memory use, or asynchronous updates undermine latency and convergence.

Latest papers

36 papers7 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks
Mar 5, 2026Burak Topcu, Musa Oguzhan Cim, Poovaiah Palangappa +2Parallel LLM Inference LLM Inference
The Pennsylvania State University, State College, Pennsylvania, USA · Advanced Micro Devices, San Jose, California, USA · Advanced Micro Devices, Portland, Oregon, USA
SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training
Mar 3, 2026Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe +5Distributed Training Fault-Tolerant Distributed Training
Pluralis Research
Orchestrating Multimodal DNN Workloads in Wireless Neural Processing
Mar 2, 2026Sai Xu, Kai-Kit Wong, Yanan Du +1Efficient Multimodal Inference Edge Inference
Department of Electronic and Electrical Engineering, University College London, WC1E 7JE, London, UK · Department of Electronic Engineering, Kyung Hee University, Yongin-si, Gyeonggi-do 17104, Republic of Korea · School of Electrical and Electronic Engineering, the University of Sheffield, Sheffield S10 2TN, UK +1
OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling
Sep 28, 2025Jihu Guo, Tenghui Ma, Wei Gao +6Communication-Efficient Distributed Training Heterogeneous Computing
FDU & Shanghai AI Laboratory · HKUST · Independent Researcher +2
Concertina: Data-Centric Adaptive Pipeline Parallelism for Efficient Heterogeneous Long-Context LLM Training
Sep 25, 2025Shiju Wang, Yujie Wang, Fangcheng Fu +6Long-Context Language Modeling Efficient Language Model Training
Tsinghua University · Peking University · Shanghai Jiao Tong University +1
Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters
Apr 7, 2025Zonghang Li, Tao Li, Wenjiao Feng +8On-Device Language Model Inference LLM Inference
MBZUAI · UESTC · City University of Hong Kong +1

Mar 5, 2026J/K move · Enter open · S save

Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks

Burak Topcu, Musa Oguzhan Cim, Poovaiah Palangappa, Meena Arunachalam+1

The Pennsylvania State University, State College, Pennsylvania, USA · Advanced Micro Devices, San Jose, California, USA · Advanced Micro Devices, Portland, Oregon, USA

Parallel LLM Inference LLM Inference Tensor Parallelism Pipeline Parallelism

Open paper AskPDF arXiv

Breakthroughs in the generative AI domain have fueled an explosion of large language model (LLM)-powered applications, whose workloads fundamentally consist of sequences of inferences through transformer architectures. Within this rapidly expanding ecosystem, dense LLMs--those that activate all model parameters for each token generation--form the foundation for advanced expert-based variants. Dense models continue to dominate because of their strong generalization ability, scalability, ease of fine-tuning, and versatility across diverse tasks. In LLM inference systems, performance is mainly characterized by latency, response time, and throughput (i.e., tokens generated per unit of time). Latency and throughput are inherently coupled: optimizing for one often comes at the expense of the other. Moreover, batching strategies and parallelism configurations, which are essential when dense model parameters exceed device memory capacity, can significantly affect both latency and overall system throughput. This paper (i) investigates the workloads of two representative dense LLMs--Llama-3.1-70B and Llama-3.1-405B, focusing in particular on intra-node parallelization schemes, (ii) analyzes how input characteristics, batching, and parallelism strategies influence latency flexibility and the latency-throughput tradeoff, and (iii) identifies key performance bottlenecks that inform design choices for meeting service-level agreements (SLAs) and sustaining inference quality. Our empirical evaluations reveal that Tensor Parallelism (TP) improves the latency objectives while Pipeline Parallelism (PP) is better-suited for throughput-oriented applications. We highlight that their hybrid usage by controlling the TP and PP degrees provides control over the latency-throughput interplay.

Latest papers

Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks

SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training

Orchestrating Multimodal DNN Workloads in Wireless Neural Processing

OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling

Concertina: Data-Centric Adaptive Pipeline Parallelism for Efficient Heterogeneous Long-Context LLM Training

Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters