Long-Context Language Modeling

Follow this topic Ask about this topic

Momentum

12 papers in the last four weeks, up 71% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 112

All topics

FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training
Jun 7, 2026Zheng Wang, Eric Liu, Linan Jiang +5Communication-Efficient Distributed Training Long-Context Language Modeling
University of California San Diego, La Jolla, USA · University of Southern California, Los Angeles, USA · Rice University, Houston, USA.
How Much Dense Attention is Necessary? Oracle-Guided Sparse Prefill for Full/GQA Layers in Hybrid Long-Context Models
Jun 5, 2026Hongxing Wang, Harenome Razanajato, Zhen Zhang +2Grouped-Query Attention Long-Context Language Modeling
Dense Contexts Are Hard Contexts: Lexical Density Limits Effective Context in LLMs
Jun 4, 2026Giovanni Dettori, Matteo Boffa, Danilo Giordano +2Long-Context Retrieval LLM Evaluation
Department of Computer Science Politecnico di Torino Torino, IT 10129 · Department of Computer Science University of Turin Torino, IT 10124
Forget Attention: Importance-Aware Attention Is All You Need
Jun 1, 2026Suhyeong Shin, Yeongwook YangSoftmax Attention Long-Context Language Modeling
Department of Computer Engineering Kangwon National University
Blurry Window Attention
May 31, 2026Axel Laborieux, Christos Sourmpis, Juan Gabriel Kostelec +1Long-Context Language Modeling Linear Attention
Huawei, Zurich, Switzerland · Huawei Advanced Computing and Storage Lab, Shenzhen, China
Don't Read Everything: A Curvature-Conditioned Query for Linear Attention
May 31, 2026Dong Le, Thong Nguyen, Cong-Duy Nguyen +1In-Context Retrieval Long-Context Language Modeling
Nanyang Technological University · National University of Singapore · VinUniversity
Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling
May 30, 2026Petr NyomaLong-Context Language Modeling State Space Models
Independent Researcher
GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs
May 29, 2026Junjie Peng, You Wu, Haoyi Wu +4KV Caching Long-Context Language Modeling
1Sun Yat-sen University · 2ShanghaiTech University · 3Guangdong Province Key Laboratory of Information Security Technology
Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation
May 28, 2026Soumyadeep Jana, Sagar Nishad, Sanasam Ranbir SinghKV Caching Long-Context Language Modeling
Indian Institute of Technology Guwahati, India
Periodic RoPE for Infinite Context LLMs
May 27, 2026Simin HuoRotary Positional Embeddings Long-Context Language Modeling
Shanghai Jiao Tong University
Why Can't They Remember? Uncovering Representation and Retrieval Bottlenecks in Multi-Turn Acoustic Memory
May 26, 2026Yang Xiao, Siyi Wang, Han Yin +4Audio-Language Model Evaluation Memorization in Language Models
1The University of Melbourne · 2KAIST · 3The University of Auckland +1
NestedKV: Nested Memory Routing for Long-Context KV Cache Compression
May 26, 2026Hong Chen, Xiang Liu, Yubo Gao +5KV Caching Long-Context Language Modeling
The Hong Kong University of Science and Technology (Guangzhou) · Jimei University
Lngram: N-gram Conditional Memory in Latent Space
May 24, 2026Yunao Zheng, Guoyang Xia, Xiaojie Wang +1Memory-Augmented Language Models Long-Context Language Modeling
Beijing University of Posts and Telecommunications (BUPT), Beijing, China · Li Auto Inc., Beijing, China
Interdomain Attention: Beyond Token-Level Key-Value Memory
May 23, 2026Naoki Kiyohara, Harrison Bo Hua Zhu, Riccardo El Hassanin +4Self-Attention Long-Context Language Modeling
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzLong-Context Language Modeling Recurrent Transformers
NVIDIA
Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention
May 19, 2026Wenhu Zhang, Yiming Wu, Huanyu Wang +6Self-Attention Long-Context Language Modeling
The Hong Kong University of Science and Technology · The University of Hong Kong · Zhejiang University +1
Elastic-dLLM: Position Preserving Context Compression and Augmentation of Diffusion LLMs
May 18, 2026Junyi Wu, Tianchen Zhao, Shaoqiu Zhang +3Masked Diffusion Models Long-Context Language Modeling
Tsinghua University · Shanghai Jiao Tong University · Infinigence AI
RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
May 15, 2026Yufeng Du, Phillip Harris, Minyang Tian +5Rotary Positional Embeddings Self-Attention
University of Illinois at Urbana-Champaign · USA · University of Bonn +3
EndPrompt: Efficient Long-Context Extension via Terminal Anchoring
May 14, 2026Han Tian, Luxuan Chen, Xinran Chen +10Long-Context Language Modeling LLM Fine-Tuning
1Nankai University · 2Baidu Inc. · 3Shanghai Jiao Tong University +1
A Hierarchical Language Model with Predictable Scaling Laws and Provable Benefits of Reasoning
May 13, 2026Jason Gaitonde, Frederic Koehler, Elchanan Mossel +2Long-Context Language Modeling Language Model Scaling Laws
Duke University · University of Chicago · Massachusetts Institute of Technology +1
Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memory
May 13, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungMemory-Augmented Neural Networks Memory-Augmented Language Models
College of Pharmacy Chungnam National University Daejeon, South Korea · Department of Computer Science & Engineering Chungnam National University Daejeon, South Korea
Training-Inference Consistent Segmented Execution for Long-Context LLMs
May 12, 2026Xianpeng Shang, Jiang Li, Zehua Duo +2Self-Attention Long-Context Language Modeling
College of Computer Science, Inner Mongolia University, Hohhot 010021, China · National & Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, Hohhot 010021, China · Inner Mongolia Key Laboratory of Multilingual Artificial Intelligence Technology, Hohhot 010021, China +1
Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing
May 11, 2026Jinchang Zhu, Jindong Li, Chengyu Zou +4Long-Context Language Modeling Adaptive Loss Weighting
1The Hong Kong University of Science and Technology (Guangzhou) · Institute of Artificial Intelligence (TeleAI), China Telecom
Mela: Test-Time Memory Consolidation based on Transformation Hypothesis
May 11, 2026Lungchuan ChenMemory-Augmented Neural Networks Memory-Augmented Language Models
MusubiAI
Remember to Forget: Gated Adaptive Positional Encoding
May 11, 2026Riccardo Ali, Alessio Borgi, Mario Severino +2Rotary Positional Embeddings Self-Attention
Department of Computer Science and Technology, University of Cambridge, United Kingdom · Department of Computer, Control and Management Engineering, Sapienza University, Italy · Department of Information Engineering, University of Padova, Italy
Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
May 11, 2026Daniel Goldstein, Eugene CheahTransformer Inference Self-Attention
Featherless AI · Eleuther AI
Positional LSH: Binary Block Matrix Approximation for Attention with Linear Biases
May 10, 2026Daniel Wolfson, Tal WagnerSelf-Attention Long-Context Language Modeling
Blavatnik School of Computer Science and AI, Tel Aviv University
VORT: Adaptive Power-Law Memory for NLP Transformers
May 9, 2026Nabil MlaikiMemory-Augmented Language Models Transformer
Department of Mathematics and Sciences, Prince Sultan University, Riyadh, Saudi Arabia
Kaczmarz Linear Attention
May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuLong-Context Language Modeling Recurrent Transformers
School of Mathematics and Statistics Xi’an Jiaotong University Xi’an, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing, China
Self-Consolidating Language Models: Continual Knowledge Incorporation from Context
May 8, 2026Zekun Wang, Anant Gupta, Zihan Dong +1Continual Learning for LLMs Long-Context Language Modeling
Georgia Institute of Technology

Jun 7, 2026, cs.DCJ/K move · Enter open · S save

FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training

Zheng Wang, Eric Liu, Linan Jiang, Zhongkai Yu+4

University of California San Diego, La Jolla, USA · University of Southern California, Los Angeles, USA · Rice University, Houston, USA.

Communication-Efficient Distributed Training Long-Context Language Modeling LLM Training

Open paper AskPDF arXiv

Context parallelism (CP) is essential for training large-scale, long-context language models, as it partitions sequences to reduce memory overhead. However, existing CP methods suffer from workload imbalance, inefficient kernels, and redundant communication due to static sequence sharding and key-value (KV) tensor communication. We present FlashCP, a load-balanced and communication-efficient framework for CP training. FlashCP introduces a sharding-aware communication mechanism to eliminate redundant KV communication and proposes a novel Whole-Doc sharding strategy that maximizes communication savings while maintaining balanced workloads. To efficiently combine Whole-Doc and Per-Doc sharding, FlashCP further designs a heuristic algorithm to search for near-optimal sharding plans. Extensive experiments show that FlashCP achieves up to 1.63x speedup over state-of-the-art CP frameworks across diverse datasets.

Momentum

Related topics

Latest papers 112