Pipeline Parallelism

Follow this topic Ask about this topic

Pipeline parallelism assigns consecutive model layers to different devices, then passes activations and gradients between stages so larger models can run across limited memory and compute. Training methods such as GPipe, 1F1B, interleaved schedules, and PipeDream trade pipeline bubbles and activation memory against synchronization and stale weights; inference systems use stage scheduling and micro-batching to balance throughput with latency. Current work adapts partitioning and schedules to heterogeneous hardware, long sequences, and mixture-of-experts models, while also overlapping communication or compressing activations. A central tension is improving device utilization without making communication, memory use, or asynchronous updates undermine latency and convergence.

Latest papers

36 papers7 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

ETH: Data-Balanced Heterogeneous Pipeline Parallelism for Training Large Models
Oct 15, 2026Mianjie Yu, Zhongmin Zhao, Zizhao Mo +9Distributed Training Pipeline Parallelism
University of Macau · Great Bay University · Sun Yat-sen University +1
Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off
Oct 14, 2026Javad Mirzaei, Jeebak MitraParallel LLM Inference LLM Inference
Office of CTO Dell Technologies Inc. Canada
BSIPipe: Bidirectional Sequence-Level Interleaved Pipeline Parallelism for Long-Sequence Large Language Model Training
Oct 10, 2026Yuzhou Ni, Ling ChenPipeline Parallelism LLM Training
State Key Laboratory of Blockchain and Data Security, Zhejiang University · College of Computer Science and Technology, Zhejiang University
Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
Oct 4, 2026Javad Mirzaei, Jeebak MitraLLM Inference Tensor Parallelism
Office of CTO Dell Technologies Inc. Canada
HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training
Sep 30, 2026Mengyuan Fan, Peizhuang Cong, Zixiao Huang +7Expert Parallelism Heterogeneous Computing
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Infinigence AI · Tsinghua University
WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse
Sep 28, 2026Aaryam SharmaLLM Inference Efficiency LLM Serving
Baseten
Spexis: Speculative Lookahead Scheduling for LLM Inference
Sep 28, 2026Hyungyu Jung, Jaehyeok Yu, Hoonseo Choi +3LLM Inference Efficiency LLM Inference Scheduling
Seoul National University · Hanyang University
Para-Pipe: Exploiting Hierarchical Operator Parallelism of ML Computational Graphs on SoCs
Sep 3, 2026Yujie Zhang, Huiying Lan, Ehsan Aghapour +5Efficient Neural Network Inference Energy-Efficient ML
School of Computing, National University of Singapore, Singapore · Parallel Computer Systems, the University of Amsterdam, 1098 XH Amsterdam, The Netherlands · Information Technology Department, Black Sesame Technologies, San Jose, CA 95131, USA
Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets
Aug 19, 2026Tate Berenbaum, Muthaiah VenkatachalamLLM Inference On-Device Language Model Inference
Not Community Labs Inc. · Intel Corporation
Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
Jul 14, 2026Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi +9Communication-Efficient Distributed Training Fault-Tolerant Distributed Training
Pluralis Research
CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems
Jul 8, 2026Tingkai Liu, Muralidhar Andoorveedu, Sanjoy Das +2GPU Kernel Optimization High-Performance Computing
University of Illinois Urbana-Champaign · NVIDIA Corporation
One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining
Jun 29, 2026Philip Zmushko, Egor Petrov, Nursultan Abdullaev +2Error Feedback Muon Optimizer
Institute of Science and Technology Austria (ISTA), Austria · Yandex, Russia · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab), Russia +2
Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation
Jun 23, 2026Sijie Wang, Zhengyu Qing, Zhiqiang Tan +6Reinforcement Learning Distributed RL
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Data Science and Analytics Thrust, The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Ltd.
Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency
Jun 5, 2026Itay Elam, Eliron Rahimi, Avi Mendelson +1Neural Network Training Dynamics Pipeline Parallelism
Department of Computer Science Technion - Israel Institute of Technology · School of Electrical and Computer Engineering Ben-Gurion University of the Negev
Learned Subspace Compression for Communication-Efficient Pipeline Parallelism
Jun 3, 2026Paul Janson, Edouard Oyallon, Eugene BelilovskyLLM Compression Communication-Efficient Distributed Training
Concordia University · Mila Quebec AI Institute · CNRS, Sorbonne University
Demystifying Pipeline Parallelism: First Theory for PipeDream
Jun 2, 2026Ivan Ilin, Peter RichtárikNonconvex Stochastic Optimization Distributed Training
KAUST
Speculative Pipeline Decoding: Higher-Accruacy and Zero-Bubble Speculation via Pipeline Parallelism
May 29, 2026Yijiong Yu, Huazheng Wang, Shuai Yuan +2Speculative Decoding LLM Inference Acceleration
aOregon State University · bDeepSolution
AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training
May 28, 2026Ling Chen, Houming Wu, Wenjie YuPipeline Parallelism LLM Training
State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China
A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training
May 19, 2026Daniel Barley, Jonathan Leis, Benjamin Klenk +1Communication-Efficient Distributed Training Pipeline Parallelism
Hardware and Artificial Intelligence (HAWAII) Lab, Heidelberg University, Heidelberg, Germany · NVIDIA Corporation, Santa Clara, CA, USA
A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability
May 18, 2026Ruitao Liu, Xinyang Tian, Shuo Chen +4High-Performance Computing Distributed Training
Tsinghua University · Scitix AI
DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism
May 10, 2026Zhichen Zeng, Chi-Chih Chang, Jiayi Wang +10Expert Parallelism Communication-Efficient Distributed Training
ByteDance Seed · University of Washington · Cornell University
Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
May 6, 2026Sajal Dash, Feiyi WangCommunication-Efficient Distributed Training High-Performance Computing
Oak Ridge National Laboratory Oak Ridge, USA
Efficient Training on Multiple Consumer GPUs with RoundPipe
Apr 29, 2026Yibin Luo, Shiwei Gao, Huichuan Zheng +2GPU Acceleration LLM Fine-Tuning
Tsinghua University, Beijing, China
Efficient, VRAM-Constrained xLM Inference on Clients
Apr 29, 2026Aditya Ukarande, Deep Shekhar, Marc Blackstein +1GPU Acceleration Efficient VLM Inference
NVIDIA Corporation, Santa Clara, CA, USA · NVIDIA Graphics Pvt Ltd., Bangalore, India
JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Training
Apr 26, 2026Zhengding Hu, Hehua Ouyang, Chang Chen +6RL Post-Training Pipeline Parallelism
University of California, San Diego
PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving
Apr 14, 2026Xu Bai, Muhammed Tawfiqul Islam, Chen Wang +1LLM Inference Efficiency LLM Serving
DisNet Lab University of Melbourne Melbourne, VIC, Australia · IBM Research Yorktown Heights, NY, USA
ResBM: Residual Bottleneck Models for Low-Bandwidth Pipeline Parallelism
Apr 13, 2026Alan Aboudib, Rodrigo Lopez Portillo A., Kalei Brady +1Activation Compression Communication-Efficient Distributed Training
Macrocosmos AI
NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining
Apr 8, 2026Zhida Jiang, Zhaolong Xing, Huichao Chai +12Distributed Training Pipeline Parallelism
Huawei
Executing as You Generate: Hiding Execution Latency in LLM Code Interpreters
Apr 1, 2026Zhensu Sun, Zhihao Lin, Zhi Chen +4LLM Inference Acceleration Pipeline Parallelism
Singapore Management University · Beihang University
Backdoor Attacks on Decentralised Post-Training
Mar 31, 2026Oğuzhan Ersoy, Nikolay Blagoev, Jona te Lintelo +3LLM Backdoor Attacks Language Model Post-Training
Gensyn · University of Neuchâtel · Radboud University +3

Oct 15, 2026J/K move · Enter open · S save

ETH: Data-Balanced Heterogeneous Pipeline Parallelism for Training Large Models

Mianjie Yu, Zhongmin Zhao, Zizhao Mo, Jianxiong Liao+8

University of Macau · Great Bay University · Sun Yat-sen University+1

Distributed Training Pipeline Parallelism Heterogeneous Computing Communication-Efficient Distributed Training

Open paper AskPDF arXiv

Current heterogeneity-aware training systems for large models balance workloads by assigning unequal numbers of layers across pipeline stages, a strategy aimed at equalizing execution time through a unique pipeline configuration. However, this approach fails to address critical memory constraints and suffers from inherently coarse balancing granularity. In this paper, we propose Heterogeneous Pipeline Parallelism (HPP), a novel approach enabling efficient and fine-grained data balance across heterogeneous pipelines. To address HPP's key challenges, prohibitive search overhead for optimal configurations and high communication overhead from asymmetric All-Reduce, we design the ETH training system. ETH employs a hierarchical workload balance framework that jointly optimizes balance across heterogeneous pipelines and within individual pipeline stages. To mitigate asymmetric communication costs, ETH implements block-wise gradient aggregation. Furthermore, it integrates specialized optimizations that dramatically reduce HPP configuration search overhead. Evaluations demonstrate ETH achieves up to 2x training speedup while reducing configuration generation overhead by up to 56x compared to state-of-the-art systems. The code is available at https://github.com/YuMJie/ETH.