Context Parallelism

Follow this topic Ask about this topic

Latest papers

10 papers1 paper in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
Oct 15, 2026Muhammad IbrahimContext Parallelism KV-Cache Management
Multiverse Computing
FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention
Oct 13, 2026Guoxia Wang, Qianyue He, Siming Wu +6Context Parallelism Attention Mechanisms
Baidu Inc.
SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
Sep 29, 2026Chuan Liu, Shuoming Zhang, Zhicheng Li +6LLM Serving Tensor Parallelism
State Key Laboratory of Processors, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training
Sep 7, 2026Zili Wang, Zhaopeng Qiu, Yuekai Zhang +2Context Parallelism Speculative Decoding
NVIDIA
Wan-Streamer v0.2: Higher Resolution, Same Latency
Jul 5, 2026Lianghua Huang, Zhi-Fan Wu, Yupeng Shi +23Audio-Video Generation Real-Time Interactive Video Generation
Wan Team, Alibaba Group
Design-CP: Context Parallelism for Design of Protein Nanoparticles
Jul 3, 2026Lorenzo Tarricone, Helen E. Eisenach, Aiko Muraishi +1GPU Acceleration High-Performance Computing
Department of Statistics, University of Oxford, Oxford, UK · Ellison Institute of Technology Oxford, Oxford, UK · Institute for Protein Design, University of Washington, Seattle, USA +2
FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training
Jun 7, 2026Zheng Wang, Eric Liu, Linan Jiang +5Communication-Efficient Distributed Training Long-Context Language Modeling
University of California San Diego, La Jolla, USA · University of Southern California, Los Angeles, USA · Rice University, Houston, USA.
Fold-CP: A Context Parallelism Framework for Biomolecular Modeling
Mar 16, 2026Dejun Lin, Simon Chu, Vishanth Iyer +35Context Parallelism Protein Structure Prediction
NVIDIA · Rezo Therapeutics · Proxima +1
Efficient Scaling of LLM Training with Flexible Context Parallelism
Feb 25, 2026Yifan Niu, Han Xiao, Dongyi Liu +2Context Parallelism Efficient Language Model Training
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co., Ltd. · The Hong Kong University of Science and Technology
Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking
Feb 24, 2026Ravi Ghadia, Maksim Abraham, Sergei Vorobyov +1Memory-Efficient Attention Context Parallelism
Together AI

Oct 15, 2026J/K move · Enter open · S save

When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill

Muhammad Ibrahim

Multiverse Computing

Context Parallelism KV-Cache Management KV-Cache Offloading LLM Inference Acceleration Disaggregated LLM Serving

Open paper AskPDF arXiv

Reusing the attention KV cache across requests is a standard way to cut prefill cost in long-context LLM serving, and production systems increasingly back the per-engine cache with a remote tier shared by every engine. We study such a tier in a disaggregated deployment of GLM-5.3, an FP8 mixture-of-experts model with multi-head latent attention (MLA), on 8xB300 nodes running prefill context parallelism (PCP8). Enabling the remote tier (LMCache with a Valkey store) left median time-to-first-token unchanged but raised the 16k-token p99 from 3.96 s to 17.3 s. Replaying a production request stream on an isolated engine, we find that a remote hit took 0.88 s against 0.31 s to recompute the prompt, and that synchronous loading turned a hit-heavy stream into an unstable queue: p99 83.32 s with the GPUs 7% busy. A two-term cost model explains both: a hit pays off only if the tier delivers more than B* = sR (2.53 GB/s for this model and GPU), and a fixed per-hit engine cost (T0 ~ 0.33 s) means that hits on prompts shorter than about 17k tokens cannot beat recompute at any bandwidth. Guided by the model, we (i) shard the MLA KV store across PCP ranks, with a proof that the existing minimum-over-ranks lookup returns the longest cached prefix and a load protocol that can never desynchronize collectives; (ii) partition the remote tier by rank, one server instance per rank with no coordination, which raises GET bandwidth 6.3x to 10.76 GB/s; (iii) add asynchronous loading to the sharded store; and (iv) fix three client bugs that caused unbounded stalls. On the hit-heavy stream, with 98.5% of prompt tokens served from the remote tier, p99 falls from 83.32 s to 0.67 s. The sharded store raises prefill throughput by 14%, and by 34% together with an upstream kernel fix. All changes are submitted upstream to LMCache.