Overview

Pipeline parallelism assigns consecutive model layers to different devices, then passes activations and gradients between stages so larger models can run across limited memory and compute. Training methods such as GPipe, 1F1B, interleaved schedules, and PipeDream trade pipeline bubbles and activation memory against synchronization and stale weights; inference systems use stage scheduling and micro-batching to balance throughput with latency. Current work adapts partitioning and schedules to heterogeneous hardware, long sequences, and mixture-of-experts models, while also overlapping communication or compressing activations. A central tension is improving device utilization without making communication, memory use, or asynchronous updates undermine latency and convergence.

Latest papers

CardsList

38 papers2 papers in the last four weeks, down 71% on the four weeks before. 0.0% of all new papers.

  1. Benchmarking LLM Inference on Desktop-Scale NVIDIA Grace Blackwell Clusters

    Nov 15, 2026Kevin Bhimani, Christian Escobar, Daniel Mihovch +2LLM InferenceParallel LLM Inference

  2. ETH: Data-Balanced Heterogeneous Pipeline Parallelism for Training Large Models

    Oct 15, 2026Mianjie Yu, Zhongmin Zhao, Zizhao Mo +9Distributed TrainingPipeline Parallelism

  3. Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

    Oct 14, 2026Javad Mirzaei, Jeebak MitraParallel LLM InferenceLLM Inference

  4. BSIPipe: Bidirectional Sequence-Level Interleaved Pipeline Parallelism for Long-Sequence Large Language Model Training

    Oct 10, 2026Yuzhou Ni, Ling ChenPipeline ParallelismLLM Training

  5. HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training

    Sep 30, 2026Mengyuan Fan, Peizhuang Cong, Zixiao Huang +7Expert ParallelismHeterogeneous Computing

  6. Spexis: Speculative Lookahead Scheduling for LLM Inference

    Sep 28, 2026Hyungyu Jung, Jaehyeok Yu, Hoonseo Choi +3LLM Inference EfficiencyLLM Inference Scheduling

  7. Para-Pipe: Exploiting Hierarchical Operator Parallelism of ML Computational Graphs on SoCs

    Sep 3, 2026Yujie Zhang, Huiying Lan, Ehsan Aghapour +5Efficient Neural Network InferenceEnergy-Efficient ML

  8. Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets

    Aug 19, 2026Tate Berenbaum, Muthaiah VenkatachalamLLM InferenceOn-Device Language Model Inference

  9. CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems

    Jul 8, 2026Tingkai Liu, Muralidhar Andoorveedu, Sanjoy Das +2GPU Kernel OptimizationHigh-Performance Computing

  10. One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining

    Jun 29, 2026Philip Zmushko, Egor Petrov, Nursultan Abdullaev +2Error FeedbackMuon Optimizer

  11. Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation

    Jun 23, 2026Sijie Wang, Zhengyu Qing, Zhiqiang Tan +6Reinforcement LearningDistributed RL

  12. Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency

    Jun 5, 2026Itay Elam, Eliron Rahimi, Avi Mendelson +1Neural Network Training DynamicsPipeline Parallelism

  13. Learned Subspace Compression for Communication-Efficient Pipeline Parallelism

    Jun 3, 2026Paul Janson, Edouard Oyallon, Eugene BelilovskyLLM CompressionCommunication-Efficient Distributed Training

  14. AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training

    May 28, 2026Ling Chen, Houming Wu, Wenjie YuPipeline ParallelismLLM Training

  15. A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training

    May 19, 2026Daniel Barley, Jonathan Leis, Benjamin Klenk +1Communication-Efficient Distributed TrainingPipeline Parallelism

  16. DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism

    May 10, 2026Zhichen Zeng, Chi-Chih Chang, Jiayi Wang +10Expert ParallelismCommunication-Efficient Distributed Training

  17. Efficient Training on Multiple Consumer GPUs with RoundPipe

    Apr 29, 2026Yibin Luo, Shiwei Gao, Huichuan Zheng +2GPU AccelerationLLM Fine-Tuning

  18. Efficient, VRAM-Constrained xLM Inference on Clients

    Apr 29, 2026Aditya Ukarande, Deep Shekhar, Marc Blackstein +1GPU AccelerationEfficient VLM Inference

  19. JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Training

    Apr 26, 2026Zhengding Hu, Hehua Ouyang, Chang Chen +6RL Post-TrainingPipeline Parallelism

  20. PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving

    Apr 14, 2026Xu Bai, Muhammed Tawfiqul Islam, Chen Wang +1LLM Inference EfficiencyLLM Serving