Context Parallelism

Follow this topic Ask about this topic

Latest papers

11 papers2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Split-KV: Efficient Context Parallelism for Training Diffusion Drafters
Nov 4, 2026Geng Zhang, Fangzhao Zhang, Cong Xie +2Context Parallelism Communication-Efficient Distributed Training
National University of Singapore · Stanford University · ByteDance Seed
When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
Oct 15, 2026Muhammad IbrahimContext Parallelism KV-Cache Management
Multiverse Computing
FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention
Oct 13, 2026Guoxia Wang, Qianyue He, Siming Wu +6Context Parallelism Attention Mechanisms
Baidu Inc.
SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
Sep 29, 2026Chuan Liu, Shuoming Zhang, Zhicheng Li +6LLM Serving Tensor Parallelism
State Key Laboratory of Processors, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training
Sep 7, 2026Zili Wang, Zhaopeng Qiu, Yuekai Zhang +4Context Parallelism Speculative Decoding
NVIDIA
Wan-Streamer v0.2: Higher Resolution, Same Latency
Jul 5, 2026Lianghua Huang, Zhi-Fan Wu, Yupeng Shi +23Audio-Video Generation Real-Time Interactive Video Generation
Wan Team, Alibaba Group
Design-CP: Context Parallelism for Design of Protein Nanoparticles
Jul 3, 2026Lorenzo Tarricone, Helen E. Eisenach, Aiko Muraishi +1GPU Acceleration High-Performance Computing
Department of Statistics, University of Oxford, Oxford, UK · Ellison Institute of Technology Oxford, Oxford, UK · Institute for Protein Design, University of Washington, Seattle, USA +2
FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training
Jun 7, 2026Zheng Wang, Eric Liu, Linan Jiang +5Communication-Efficient Distributed Training Long-Context Language Modeling
University of California San Diego, La Jolla, USA · University of Southern California, Los Angeles, USA · Rice University, Houston, USA.
Fold-CP: A Context Parallelism Framework for Biomolecular Modeling
Mar 16, 2026Dejun Lin, Simon Chu, Vishanth Iyer +35Context Parallelism Protein Structure Prediction
NVIDIA · Rezo Therapeutics · Proxima +1
Efficient Scaling of LLM Training with Flexible Context Parallelism
Feb 25, 2026Yifan Niu, Han Xiao, Dongyi Liu +2Context Parallelism Efficient Language Model Training
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co., Ltd. · The Hong Kong University of Science and Technology
Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking
Feb 24, 2026Ravi Ghadia, Maksim Abraham, Sergei Vorobyov +1Memory-Efficient Attention Context Parallelism
Together AI

Nov 4, 2026J/K move · Enter open · S save

Split-KV: Efficient Context Parallelism for Training Diffusion Drafters

Geng Zhang, Fangzhao Zhang, Cong Xie, Tiantian Fan+1

National University of Singapore · Stanford University · ByteDance Seed

Context Parallelism Communication-Efficient Distributed Training Speculative Decoding for Diffusion Language Models

Open paper AskPDF arXiv

Diffusion drafters accelerate speculative decoding by denoising multiple draft tokens in parallel. Their training uses anchor sampling to construct compact draft queries and performs cross-attention with a block-sparse mask over a much longer context. We empirically show that acceptance length remains stable as context scales with a fixed query budget, while increasing the query budget yields diminishing quality gains, jointly establishing a practical regime with queries far shorter than the context. No existing context parallelism (CP) method jointly exploits this query--context asymmetry and meets the nonlinear, sample-dependent load-balancing requirement created by anchor sampling. We present Split-KV, an exact CP method that keeps key/value shards local and limits per-layer attention communication to query-sized tensors, making this communication independent of context length. Its global query view further enables a cost-aware longest-processing-time-first (LPT) strategy that balances context key/value chunks by their realized attention cost. At a CP size of 64, Split-KV retains 85--91% weak-scaling efficiency and achieves up to

3.67\times

speedup over the fastest available CP baseline. LPT achieves

<

1% imbalance and plans three orders of magnitude faster than the best load-balancing baseline. Code is available at https://github.com/zxgx/verl.