cs.DCFeb 25, 2026

Efficient Scaling of LLM Training with Flexible Context Parallelism

Authors: Yifan Niu, Han Xiao, Dongyi Liu, Wei Zhou, Jia Li

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co., Ltd. · The Hong Kong University of Science and Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Scaling long-context capabilities is crucial for Large Language Models (LLMs). However, real-world data contain a large number of sequences with heterogeneous lengths. Existing training libraries for LLMs rely on static parallelism strategies, which suffer from severe load imbalance, redundant communication, and suboptimal hardware utilization under data heterogeneity. In this work, we propose Flexible Context Parallelism (FCP), an efficient parallelism strategy that adaptively reconfigures communication groups and context parallelism degrees during LLM training. We generalize more flexible non-power-of-two parallelism degrees and develop a polynomial-time algorithm to generate near-optimal parallelism strategies with only millisecond-level overhead per training batch. FCP is able to maintain high hardware efficiency even under extreme data heterogeneity. Experimental results demonstrate that FCP significantly outperforms Megatron-LM and DeepSpeed in both LLM and MLLM training, achieving up to 1.46x speedup in average throughput while maintaining near-linear scaling efficiency across large-scale clusters. For extremely unbalanced batches, FCP even achieves 2.24x speedup.

Figures & tables

Figure 1: (a) Length distribution of text and multimodal datasets. (b) Compared to static parallelism, dynamic parallelism facilitates load balancing against heterogeneous data.
Figure 2: Overall workflow of FCP.
Figure 3: Average iteration time (in seconds) comparison on text datasets for Qwen3-1.7B/4B/8B, Llama3.2-1B/3B, and Llama3-8B. We compare Megatron-LM (diagonal stripes), DeepSpeed (dots), and FCP (vertical stripes) on GitHub, CommonCrawl, and Wikipedia.
Figure 4: Average iteration time (in seconds) comparison for unfrozen MLLM training across MSRVTT, InternVid, and OpenVid datasets. The evaluated models include InternVL3-2B/8B, InternVL2.5-4B and Qwen3VL-2B/4B/8B.
Figure 5: Token throughput (in k tokens/s) comparison across different cluster size (8, 16, 32, and 64) for FCP, DeepSpeed, and Megatron-LM.
ModelModel Parameters
 2B4B8B
Qwen3VL7.937.936.716.714.274.27
InternVL2.5/37.487.486.546.544.124.12
Table 3: Time cost estimation error (%).
Case StudyCase 1Case 2
DeepSpeed⟨8⟩×4\langle 8\rangle\times 4⟨4⟩×8\langle 4\rangle\times 8
Megatron-LM⟨8⟩×4\langle 8\rangle\times 4⟨4⟩×8\langle 4\rangle\times 8
FCP⟨8⟩×1\langle 8\rangle\times 1 ⟨6⟩×2\langle 6\rangle\times 2 ⟨4⟩×1\langle 4\rangle\times 1 ⟨2⟩×2\langle 2\rangle\times 2 ⟨1⟩×4\langle 1\rangle\times 4⟨4⟩×2\langle 4\rangle\times 2 ⟨3⟩×4\langle 3\rangle\times 4 ⟨2⟩×6\langle 2\rangle\times 6
Table 6: CP groups assigned by FCP.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model#Layers#Heads#GroupsHidden DimFFN Hidden Dim
Qwen3-1.7B2816820486144
Qwen3-4B3632825609728
Qwen3-8B36328409612288
Llama3.2-1B1632820488192
Llama3.2-3B2824830728192
Llama3-8B32328409614336
Appendix
Table 7: LLM models for evaluation.
Model#Layers#Heads#GroupsHidden DimVision Hidden Dim
InternVL3-2B2812215361024
InternVL2.5-4B3616220481024
InternVL3-8B2828435841024
Qwen3-VL-2B2816820481024
Qwen3-VL-4B3632825601024
Qwen3-VL-8B3632840961152
Appendix
Table 8: MLLM models for evaluation.
Figure 6: Average iteration time (in seconds) comparison for frozen-vision MLLM training across MSRVTT, InternVid, and OpenVid datasets. FCP keeps the same scheduling mechanism while the vision encoder is frozen, yielding shorter wall time than the unfrozen setting.
More in this topic

Context Parallelism

See all 10 papers

Explore similar work

CardsList

  1. FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training

    Jun 7, 2026Zheng Wang, Eric Liu, Linan Jiang +5Communication-Efficient Distributed TrainingLong-Context Language Modeling

    University of California San Diego, La Jolla, USA · University of Southern California, Los Angeles, USA · Rice University, Houston, USA.

  2. Concertina: Data-Centric Adaptive Pipeline Parallelism for Efficient Heterogeneous Long-Context LLM Training

    Sep 25, 2025Shiju Wang, Yujie Wang, Fangcheng Fu +6Long-Context Language ModelingEfficient Language Model Training

    Tsinghua University · Peking University · Shanghai Jiao Tong University +1

  3. AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism

    Apr 29, 2026Ahan Gupta, Zhihao Wang, Neel Dani +3Efficient Language Model TrainingLLM Training

    SSAIL Lab, University of Illinois Urbana-Champaign Champaign, IL, USA · Anyscale San Francisco, CA, USA · Snowflake San Mateo, CA, USA