ETH: Data-Balanced Heterogeneous Pipeline Parallelism for Training Large Models

Mianjie Yu; Zhongmin Zhao; Zizhao Mo; Jianxiong Liao; Wenxuan Li; Yulin Qiao; Huanle Xu; Wenquan Yang

cs.DCOct 15, 2026

ETH: Data-Balanced Heterogeneous Pipeline Parallelism for Training Large Models

Authors: Mianjie Yu, Zhongmin Zhao, Zizhao Mo, Jianxiong Liao, Wenxuan Li, Yulin Qiao, Huanle Xu, Wenquan Yang,

Organizations: University of Macau · Great Bay University · Sun Yat-sen University · Independent Researchers

Abstract

Current heterogeneity-aware training systems for large models balance workloads by assigning unequal numbers of layers across pipeline stages, a strategy aimed at equalizing execution time through a unique pipeline configuration. However, this approach fails to address critical memory constraints and suffers from inherently coarse balancing granularity. In this paper, we propose Heterogeneous Pipeline Parallelism (HPP), a novel approach enabling efficient and fine-grained data balance across heterogeneous pipelines. To address HPP's key challenges, prohibitive search overhead for optimal configurations and high communication overhead from asymmetric All-Reduce, we design the ETH training system. ETH employs a hierarchical workload balance framework that jointly optimizes balance across heterogeneous pipelines and within individual pipeline stages. To mitigate asymmetric communication costs, ETH implements block-wise gradient aggregation. Furthermore, it integrates specialized optimizations that dramatically reduce HPP configuration search overhead. Evaluations demonstrate ETH achieves up to 2x training speedup while reducing configuration generation overhead by up to 56x compared to state-of-the-art systems. The code is available at https://github.com/YuMJie/ETH.

Figures & tables

Figure 1: Layer-balancing versus data-balancing.

Figure 2: An example of HPP configuration. HPP not only supports the training configuration with independent pipelines that have different local batch size, micro-batch size, but also supports the unique stage configurations that have different devices, layers and parallelism strategies.

Figure 3: The latency gap under different layer balancing methods across various models. We compare maximum-to-minimum device execution latency ratios for four approaches: a) HexiScale’s layer-balancing ( HexiScale ), b) Metis’ layer-balancing ( Metis ), c) Metis without memory constraints ( Unlimited Memory ), and d) ideal balance ( Ideal ).

Panel of Figure 4 — Figure 4: The advantage of data balance and the high search overhead for HPP.

Figure 5: Asymmetric All-Reduce across four devices. Device-wise All-Reduce pads the missing gradient to ensure consistency, but brings the padding overhead. Tensor-wise All-Reduce splits the gradient by operator tensor.

Figure 6: Overview of ETH system.

Figure 7: The workflow of Block-wise All-Reduce. ETH ensures address consistency and size consistency through partition and combination, then reduce the connections by merging the gradient block which belong to the same communication group.

Panel of Figure 8 — Figure 8: Throughput sensitivity of heterogeneous pipeline configurations to $V_{inter}$ and $V_{intra}$ .

Clusters	Comp. Power	Memory	Bandwidth
Ascend 910B2	376 TFLOPS	64 GB	1536 Gbps
Ascend 910B3	313 TFLOPS	64 GB	1536 Gbps
Ascend 910B4	280 TFLOPS	32 GB	736 Gbps
NVIDIA A100	312 TFLOPS	80 GB	1536 Gbps
NVIDIA RTX3090	80 TFLOPS	24 GB	1408 Gbps

Table 1: The configurations for heterogeneous accelerators

Model	#Layer	Hidden Dim	Clusters Scales
350M	24	1024	1 $\times$ 2 $\times$ $A_{i}$ + 1 $\times$ 2 $\times$ $A_{j}$
1.3B	24	2048	2 $\times$ 2 $\times$ $A_{i}$ + 2 $\times$ 2 $\times$ $A_{j}$
2.6B	32	2560	2 $\times$ 4 $\times$ $A_{i}$ + 2 $\times$ 4 $\times$ $A_{j}$
6.7B	32	4096	4 $\times$ 4 $\times$ $A_{i}$ + 4 $\times$ 4 $\times$ $A_{j}$
13B	40	5120	4 $\times$ 8 $\times$ $A_{i}$ + 4 $\times$ 8 $\times$ $A_{j}$
70B	80	8192	32 $\times$ 8 $\times$ $A_{i}$ + 32 $\times$ 8 $\times$ $A_{j}$

Table 2: The various models and corresponding cluster scale.

N\times M

represents

N

nodes with

M

devices for each node,

A_{i}

and

A_{j}

present different accelerators

Figure 9: Average training time per iteration and generation time in different models and heterogeneous clusters.

Panel of Figure 10 — Figure 10: The performance of three All-Reduce mechanisms and the breakdown of iteration time of different methods in various models on $B2$ / $B4$ heterogeneous clusters.

Panel of Figure 11 — Figure 11: Scalability and global-search evaluation. (a)-(c) Performance comparison on GPT-3 13B when varying device count, GBS, and sequence length on $B2$ / $B4$ clusters. (d) TypeSplit-HPP comparison across GPT-3 models on A100/RTX 3090 clusters.

Panel of Figure 12 — Figure 12: Memory efficiency and predictor accuracy across GPT-3 models on A100/RTX 3090 clusters. w/o MO disables ETH’s memory-saving strategies; positive and negative errors denote overestimation and underestimation, respectively.

Panel of Figure 13 — Figure 13: Search-optimization evaluation for GPT-3 13B on $B2$ / $B4$ clusters: (a) cumulative ablation and (b) sensitivity to tolerance factor $\alpha$ . CPC : Caching Pipeline Configuration; SES : Search via Early Skipping; OSO : Optimization for Search Order; DPO : Dynamic Programming Optimization.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Metis
Pipeline	Stage	Dev.	TP/DP	#L	Recomp. Operator
Mbs=4 Gbs=1024	0	$B2\times 8$	2/4	5	None
	1	$B2\times 8$	2/4	6	None
	2	$B2\times 8$	4/2	7	None
	3	$B2\times 8$	2/4	6	None
	4	$B4\times 8$	8/1	3	None

Appendix

Table 3: Visualization of parallelization strategy generated by Metis, HexiScale and ETH. #L: number of layers in each stage. Qkv : Query, key and value. Sco : QK Score. AS : Attention Softmax. AD : Attention Dropout

Figure 14: Loss curves of ETH, HexiScale, and Metis on heterogeneous clusters, together with ETH on a homogeneous cluster. The right panel zooms in on the final iterations.

Distributed Training

Distributed training coordinates computation across multiple devices or machines to train models that exceed single-device limits.

See all 129 papers

AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training
May 28, 2026Ling Chen, Houming Wu, Wenjie YuPipeline Parallelism LLM Training
State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China
OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling
Sep 28, 2025Jihu Guo, Tenghui Ma, Wei Gao +6Communication-Efficient Distributed Training Heterogeneous Computing
FDU & Shanghai AI Laboratory · HKUST · Independent Researcher +2
Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency
Jun 5, 2026Itay Elam, Eliron Rahimi, Avi Mendelson +1Neural Network Training Dynamics Pipeline Parallelism
Department of Computer Science Technion - Israel Institute of Technology · School of Electrical and Computer Engineering Ben-Gurion University of the Negev

May 28, 2026J/K move · Enter open · S save

AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training

Ling Chen, Houming Wu, Wenjie Yu

State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China

Pipeline Parallelism LLM Training

Open paper AskPDF arXiv

Pipeline parallelism is essential for large-scale model training, but existing asynchronous approaches often degrade convergence due to parameter mismatch between forward and backward passes. We propose Asynchronous Multi-Directional Pipeline parallelism (AMDP) to mitigate this issue while sustaining high utilization. AMDP limits the first stage of each pipeline to process at most two minibatches before backpropagation, bounding the number of parameter updates between forward and backward passes. To alleviate the resulting pipeline bubbles, AMDP launches multiple concurrent pipelines and adapts their number according to pipeline depth. In addition, AMDP accumulates gradients across minibatches and applies them in a single update, ensuring that only a bounded number of minibatches experience parameter mismatch, limited to within one optimization step. Experiments on GPT- and BERT-style models demonstrate that AMDP significantly accelerates training while preserving convergence.

Abstract

Distributed Training

Explore similar work

AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training

OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling

Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency