BSIPipe: Bidirectional Sequence-Level Interleaved Pipeline Parallelism for Long-Sequence Large Language Model Training

Yuzhou Ni; Ling Chen

cs.LGOct 10, 2026

BSIPipe: Bidirectional Sequence-Level Interleaved Pipeline Parallelism for Long-Sequence Large Language Model Training

Authors: Yuzhou Ni, Ling Chen

Organizations: State Key Laboratory of Blockchain and Data Security, Zhejiang University · College of Computer Science and Technology, Zhejiang University

Abstract

Pipeline parallelism has become a critical technique for training large language models. To address the throughput limitation of existing sequence-level methods that schedule subsequences unidirectionally, we propose bidirectional sequence-level interleaved pipeline parallelism (BSIPipe), which co-schedules two opposing V-shaped interleaved pipelines at the subsequence level. To bound the activation accumulation from subsequence scaling, BSIPipe introduces a basic-unit concatenation mechanism that keeps activation memory invariant to the subsequence count. To eliminate communication deadlock risk from bidirectional scheduling, BSIPipe employs a communication optimization strategy that combines topology-aware device mapping with communication-computation overlap. In addition, BSIPipe composes with tensor parallelism to overcome the memory limitation of single-dimensional parallelism, neutralizing its bidirectional memory overhead while maintaining throughput. Experiments on GPT and LLaMA3 demonstrate that BSIPipe achieves average speedups of 1.11 $\times$ to 1.13 $\times$ in long-sequence training over SOTA sequence-level baselines.

Figures & tables

Figure 1: Comparison of representative sequence-level pipeline parallelism methods. (a) TeraPipe schedules subsequences in a forward-then-backward manner. (b) Seq1F1B applies the 1F1B schedule at the subsequence level. (c) Seq1F1B-I further combines this with model chunk interleaving, where shades of each color denote computations of different model chunks on the same device. (d) BSIPipe (ours) co-schedules two opposing V-shaped interleaved pipelines at the subsequence level.

Method	Bubble Ratio	Activation	Parameter
TeraPipe	$\frac{w-1}{N_{s}+w-1}$	$N_{s}M_{a}$	$M_{\Theta}$
Seq1F1B	$\frac{w-1}{N_{s}+w-1}$	$[M_{a},wM_{a}]$	$M_{\Theta}$
Seq1F1B-I	$\frac{w-1}{2N_{s}+w-1}$	$[\tfrac{w+1}{2}M_{a},wM_{a}]$	$M_{\Theta}$
BSIPipe (Ours)	$\frac{w-1}{3N_{s}+w-1}$	$[(w+3)M_{a},2wM_{a}]$	$2M_{\Theta}$

Table 1: Comparison of bubble ratio and per-device memory across sequence-level pipeline parallelism methods.

Panel of Figure 2 — Figure 2: Scaling BSIPipe along the subsequence dimension.

Figure 3: BSIPipe device mapping under multi-dimensional parallelism. Each node holds two TP groups (one per pipeline) with mirrored-pair gradient-synchronization groups. Dashed arrows trace the bidirectional pipeline data path.

Method	GPT-1.3B							LLaMA3-1.3B
	Sequence length				GPU count			Sequence length				GPU count
	5K	10K	15K	20K	8	16	32	5K	10K	15K	20K	8	16	32
DAPPLE (PPoPP 2021)	81,823	73,190	66,487	57,233	32,625	57,233	81,456	91,126	80,097	69,780	61,438	35,282	61,438	97,101
1F1B-I (SC 2021)	85,826	77,698	68,855	OOM	OOM	OOM	OOM	95,155	84,876	74,117	OOM	OOM	OOM	OOM
Chimera (SC 2021)	95,157	80,222	68,344	58,928	31,296	58,928	83,625	109,155	91,081	75,881	64,431	34,409	64,431	108,599
TeraPipe (ICML 2021)	48,404	46,825	41,993	37,722	15,322	37,722	38,915	57,606	52,884	46,404	40,932	22,743	40,932	32,386

Table 2: Training throughput (tokens/s) of BSIPipe and baselines. The sequence length block is evaluated on 16 GPUs with 64 micro-batches under varying sequence length. The GPU count block is evaluated at a sequence length of 20K with the number of micro-batches scaled proportionally to the GPU count. The best results are bolded . The second-best results are underlined .

Model	$\mathbf{(16,1)}$		$\mathbf{(8,2)}$		$\mathbf{(4,4)}$
Model	Tput	Memory	Tput	Memory	Tput	Memory
GPT	68,954	49.8–60.1	69,131	28.3–31.7	63,917	17.1–18.4
LLaMA3	73,453	55.8–67.8	74,541	31.4–35.5	68,565	19.4–19.7

Table 3: Training throughput (Tput, tokens/s) and per-device peak memory range (GiB) of BSIPipe under different

(w,D_{t})

configurations on 16 GPUs at a sequence length of 20K. The best results are bolded .

Figure 4: Memory footprint distributions (GiB) of BSIPipe and baselines. (a, b) Per-device peak memory under varying sequence length on 16 GPUs. (c, d) Per-device peak memory under varying GPU count at a sequence length of 20K. Method abbreviations: D = DAPPLE, I = 1F1B-I, Ch = Chimera, T = TeraPipe, S = Seq1F1B, SI = Seq1F1B-I, and B = BSIPipe.

Setting	Throughput		Drop vs. baseline
Setting	GPT	LLaMA3	GPT	LLaMA3
BSIPipe (baseline)	68,954	73,453	—	—
w/o device mapping	67,861	72,780	1.59%	0.92%
w/o F-B overlap	68,702	73,015	0.37%	0.60%

Table 4: Ablation of communication optimizations on 16 GPUs at a sequence length of 20K. Throughput in tokens/s. F-B overlap: forward-communication and backward-computation overlap.

Pipeline Parallelism

Pipeline parallelism splits neural networks into stages across devices and overlaps their execution to scale training and inference.

See all 36 papers

BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training
Sep 2, 2026Bigyan Ghimire, Jon C. CalhounCommunication-Efficient Distributed Training Long-Context Language Modeling
Holcombe Department of Electrical and Computer Engineering Clemson University Clemson, USA
AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training
May 28, 2026Ling Chen, Houming Wu, Wenjie YuPipeline Parallelism LLM Training
State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China
Concertina: Data-Centric Adaptive Pipeline Parallelism for Efficient Heterogeneous Long-Context LLM Training
Sep 25, 2025Shiju Wang, Yujie Wang, Fangcheng Fu +6Long-Context Language Modeling Efficient Language Model Training
Tsinghua University · Peking University · Shanghai Jiao Tong University +1

Sep 2, 2026J/K move · Enter open · S save

BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training

Bigyan Ghimire, Jon C. Calhoun

Holcombe Department of Electrical and Computer Engineering Clemson University Clemson, USA

Communication-Efficient Distributed Training Long-Context Language Modeling LLM Training

Open paper AskPDF arXiv

Long-context reasoning for large language models (LLMs) is becoming increasingly important, but training over long sequences remains challenging due to massive memory and communication requirements. Sequence parallelism has emerged as an essential technique for addressing bottlenecks in long sequence LLM training. However, we observe that existing sequence parallelism methods are batch-agnostic and apply uniform sequence partitioning across all batch sizes, resulting in inefficient communication. In this paper, we introduce Batch- Aware Sequence Parallelism (BASP), a sequence parallelism approach that leverages batch structure to reduce communication overhead. BASP exploits batch structure by partitioning GPUs into disjoint sequence-parallel groups according to the micro- batch size. This design reduces the all-to-all communication group size, thereby localizing communication and improving training efficiency. Experimental results on an NVIDIA A100 cluster show that BASP improves end-to-end training time by up to 1.17 - 1.31x in Llama and Qwen models compared to standard sequence parallel baselines, while preserving identical model accuracy and memory usage.

Abstract

Pipeline Parallelism

Explore similar work

BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training

AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training

Concertina: Data-Centric Adaptive Pipeline Parallelism for Efficient Heterogeneous Long-Context LLM Training