Current heterogeneity-aware training systems for large models balance workloads by assigning unequal numbers of layers across pipeline stages, a strategy aimed at equalizing execution time through a unique pipeline configuration. However, this approach fails to address critical memory constraints and suffers from inherently coarse balancing granularity. In this paper, we propose Heterogeneous Pipeline Parallelism (HPP), a novel approach enabling efficient and fine-grained data balance across heterogeneous pipelines. To address HPP's key challenges, prohibitive search overhead for optimal configurations and high communication overhead from asymmetric All-Reduce, we design the ETH training system. ETH employs a hierarchical workload balance framework that jointly optimizes balance across heterogeneous pipelines and within individual pipeline stages. To mitigate asymmetric communication costs, ETH implements block-wise gradient aggregation. Furthermore, it integrates specialized optimizations that dramatically reduce HPP configuration search overhead. Evaluations demonstrate ETH achieves up to 2x training speedup while reducing configuration generation overhead by up to 56x compared to state-of-the-art systems. The code is available at https://github.com/YuMJie/ETH.
Figures & tables
Figure 1: Layer-balancing versus data-balancing.
Figure 2: An example of HPP configuration. HPP not only supports the training configuration with independent pipelines that have different local batch size, micro-batch size, but also supports the unique stage configurations that have different devices, layers and parallelism strategies.
Figure 3: The latency gap under different layer balancing methods across various models. We compare maximum-to-minimum device execution latency ratios for four approaches: a) HexiScale’s layer-balancing ( HexiScale ), b) Metis’ layer-balancing ( Metis ), c) Metis without memory constraints ( Unlimited Memory ), and d) ideal balance ( Ideal ).
Figure 4: The advantage of data balance and the high search overhead for HPP.
Figure 5: Asymmetric All-Reduce across four devices. Device-wise All-Reduce pads the missing gradient to ensure consistency, but brings the padding overhead. Tensor-wise All-Reduce splits the gradient by operator tensor.
Figure 6: Overview of ETH system.
Figure 7: The workflow of Block-wise All-Reduce. ETH ensures address consistency and size consistency through partition and combination, then reduce the connections by merging the gradient block which belong to the same communication group.
Figure 8: Throughput sensitivity of heterogeneous pipeline configurations to Vinter and Vintra .
Clusters
Comp. Power
Memory
Bandwidth
Ascend 910B2
376 TFLOPS
64 GB
1536 Gbps
Ascend 910B3
313 TFLOPS
64 GB
1536 Gbps
Ascend 910B4
280 TFLOPS
32 GB
736 Gbps
NVIDIA A100
312 TFLOPS
80 GB
1536 Gbps
NVIDIA RTX3090
80 TFLOPS
24 GB
1408 Gbps
Table 1: The configurations for heterogeneous accelerators
Model
#Layer
Hidden Dim
Clusters Scales
350M
24
1024
1 × 2 ×Ai + 1 × 2 ×Aj
1.3B
24
2048
2 × 2 ×Ai + 2 × 2 ×Aj
2.6B
32
2560
2 × 4 ×Ai + 2 × 4 ×Aj
6.7B
32
4096
4 × 4 ×Ai + 4 × 4 ×Aj
13B
40
5120
4 × 8 ×Ai + 4 × 8 ×Aj
70B
80
8192
32 × 8 ×Ai + 32 × 8 ×Aj
Table 2: The various models and corresponding cluster scale. N×M represents N nodes with M devices for each node, Ai and Aj present different accelerators
Figure 9: Average training time per iteration and generation time in different models and heterogeneous clusters.
Figure 10: The performance of three All-Reduce mechanisms and the breakdown of iteration time of different methods in various models on B2 / B4 heterogeneous clusters.
Figure 11: Scalability and global-search evaluation. (a)-(c) Performance comparison on GPT-3 13B when varying device count, GBS, and sequence length on B2 / B4 clusters. (d) TypeSplit-HPP comparison across GPT-3 models on A100/RTX 3090 clusters.
Figure 12: Memory efficiency and predictor accuracy across GPT-3 models on A100/RTX 3090 clusters. w/o MO disables ETH’s memory-saving strategies; positive and negative errors denote overestimation and underestimation, respectively.
Figure 13: Search-optimization evaluation for GPT-3 13B on B2 / B4 clusters: (a) cumulative ablation and (b) sensitivity to tolerance factor α . CPC : Caching Pipeline Configuration; SES : Search via Early Skipping; OSO : Optimization for Search Order; DPO : Dynamic Programming Optimization.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Metis
Pipeline
Stage
Dev.
TP/DP
#L
Recomp. Operator
Mbs=4 Gbs=1024
0
B2×8
2/4
5
None
1
B2×8
2/4
6
None
2
B2×8
4/2
7
None
3
B2×8
2/4
6
None
4
B4×8
8/1
3
None
Appendix
Table 3: Visualization of parallelization strategy generated by Metis, HexiScale and ETH. #L: number of layers in each stage. Qkv : Query, key and value. Sco : QK Score. AS : Attention Softmax. AD : Attention Dropout
Figure 14: Loss curves of ETH, HexiScale, and Metis on heterogeneous clusters, together with ETH on a homogeneous cluster. The right panel zooms in on the final iterations.
Pipeline parallelism is essential for large-scale model training, but existing asynchronous approaches often degrade convergence due to parameter mismatch between forward and backward passes. We propose Asynchronous Multi-Directional Pipeline parallelism (AMDP) to mitigate this issue while sustaining high utilization. AMDP limits the first stage of each pipeline to process at most two minibatches before backpropagation, bounding the number of parameter updates between forward and backward passes. To alleviate the resulting pipeline bubbles, AMDP launches multiple concurrent pipelines and adapts their number according to pipeline depth. In addition, AMDP accumulates gradients across minibatches and applies them in a single update, ensuring that only a bounded number of minibatches experience parameter mismatch, limited to within one optimization step. Experiments on GPT- and BERT-style models demonstrate that AMDP significantly accelerates training while preserving convergence.
Ling Chen, Houming Wu, Wenjie Yu
State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China
Pipeline parallelism is widely used to train large language models (LLMs). However, increasing heterogeneity in model architectures exacerbates pipeline bubbles, thereby reducing training efficiency. Prior approaches typically optimize a single phase of the pipeline schedule (i.e., partitioning, placement, or scheduling), leaving substantial pipeline bubbles. While promising, co-optimization poses three key challenges: (1) complex performance modeling, (2) a combinatorial search space, and (3) irregular execution orders. To address these challenges, we propose OctoPipe, a pipeline parallelism system to jointly optimize partitioning, placement, and scheduling. First, we build a graph-based pipeline simulator to model heterogeneous pipeline execution for co-optimization. Second, on top of the simulator, we develop an iterative bubble-aware tuner to efficiently explore the combinatorial search space. Third, we implement a unified pipeline executor that dynamically orchestrates computation and communication to support irregular execution orders without deadlocks while maximizing communication-computation overlap. Experiments show that OctoPipe achieves 1.09--1.49× throughput improvement over the state-of-the-art pipeline parallelism approaches across various heterogeneous model configurations and GPU cluster scales.
Pipeline parallelism is essential for training large neural networks, but existing schedules trade off throughput, memory, and optimization consistency. Synchronous pipelines preserve forward/backward weight consistency but suffer from bubbles; asynchronous pipelines remove bubbles but introduce weight-version mismatch, typically requiring weight stashing, prediction, or correction mechanisms. We introduce PACI (Pipeline Asynchronous training with Controlled Inconsistency), a bubble-free asynchronous pipeline method that bounds forward/backward version drift without weight stashing, prediction, additional parameter copies, or global synchronization. The key idea is to use local gradient accumulation as a version-control mechanism: by slowing parameter-version evolution relative to pipeline delay, PACI limits the number of optimizer updates crossed by any micro-batch while preserving steady-state utilization. In GPT-style language-model pretraining, PACI matches the stability and final perplexity of synchronous 1F1B-flush, retains the same peak memory footprint, achieves fully utilized pipeline throughput, and improves training time-to-accuracy by up to 1.69× over the fastest flush baseline. These results show that forward/backward inconsistency need not be eliminated: when explicitly bounded, it can be safely traded for substantial efficiency gains.
Itay Elam, Eliron Rahimi, Avi Mendelson +1
Department of Computer Science Technion - Israel Institute of Technology · School of Electrical and Computer Engineering Ben-Gurion University of the Negev