Long-Context Language Modeling

Follow this topic Ask about this topic

The field asks how models can retrieve relevant evidence and maintain useful context as sequences grow, for tasks such as document understanding, code analysis, and long-range recall. Approaches include sparse and hierarchical attention, positional-encoding extensions, recurrent and state-space memory, hybrid architectures, and compression of context or key-value caches. Training methods range from long-sequence continued pretraining and curricula to distillation and test-time adaptation, with evaluation often using retrieval and reasoning benchmarks such as RULER, LongBench, and Needle-in-a-Haystack. The central tension is reducing compute and memory without losing reliable access to distant information; performance also depends on information density and may not extrapolate from short training contexts.

Latest papers

189 papers38 papers in the last four weeks, up 90% on the four weeks before. 0.2% of all new papers.

Interdomain Attention: Beyond Token-Level Key-Value Memory
May 23, 2026Naoki Kiyohara, Harrison Bo Hua Zhu, Riccardo El Hassanin +4Self-Attention Long-Context Language Modeling
Imperial College London, UK · Canon Inc., Japan · University of Copenhagen, Denmark +2
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzLong-Context Language Modeling Recurrent Transformers
NVIDIA
Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention
May 19, 2026Wenhu Zhang, Yiming Wu, Huanyu Wang +6Self-Attention Long-Context Language Modeling
The Hong Kong University of Science and Technology · The University of Hong Kong · Zhejiang University +1
Elastic-dLLM: Position Preserving Context Compression and Augmentation of Diffusion LLMs
May 18, 2026Junyi Wu, Tianchen Zhao, Shaoqiu Zhang +3Masked Diffusion Models Long-Context Language Modeling
Tsinghua University · Shanghai Jiao Tong University · Infinigence AI
RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
May 15, 2026Yufeng Du, Phillip Harris, Minyang Tian +5Rotary Positional Embeddings Self-Attention
University of Illinois at Urbana-Champaign · USA · University of Bonn +3
EndPrompt: Efficient Long-Context Extension via Terminal Anchoring
May 14, 2026Han Tian, Luxuan Chen, Xinran Chen +10Long-Context Language Modeling LLM Fine-Tuning
1Nankai University · 2Baidu Inc. · 3Shanghai Jiao Tong University +1
A Hierarchical Language Model with Predictable Scaling Laws and Provable Benefits of Reasoning
May 13, 2026Jason Gaitonde, Frederic Koehler, Elchanan Mossel +2Long-Context Language Modeling Language Model Scaling Laws
Duke University · University of Chicago · Massachusetts Institute of Technology +1
Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memory
May 13, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungMemory-Augmented Neural Networks Memory-Augmented Language Models
College of Pharmacy Chungnam National University Daejeon, South Korea · Department of Computer Science & Engineering Chungnam National University Daejeon, South Korea
Training-Inference Consistent Segmented Execution for Long-Context LLMs
May 12, 2026Xianpeng Shang, Jiang Li, Zehua Duo +2Self-Attention Long-Context Language Modeling
College of Computer Science, Inner Mongolia University, Hohhot 010021, China · National & Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, Hohhot 010021, China · Inner Mongolia Key Laboratory of Multilingual Artificial Intelligence Technology, Hohhot 010021, China +1
Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing
May 11, 2026Jinchang Zhu, Jindong Li, Chengyu Zou +4Long-Context Language Modeling Adaptive Loss Weighting
1The Hong Kong University of Science and Technology (Guangzhou) · Institute of Artificial Intelligence (TeleAI), China Telecom
Mela: Test-Time Memory Consolidation based on Transformation Hypothesis
May 11, 2026Lungchuan ChenMemory-Augmented Neural Networks Memory-Augmented Language Models
MusubiAI
Remember to Forget: Gated Adaptive Positional Encoding
May 11, 2026Riccardo Ali, Alessio Borgi, Mario Severino +2Rotary Positional Embeddings Self-Attention
Department of Computer Science and Technology, University of Cambridge, United Kingdom · Department of Computer, Control and Management Engineering, Sapienza University, Italy · Department of Information Engineering, University of Padova, Italy
Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
May 11, 2026Daniel Goldstein, Eugene CheahTransformer Inference Self-Attention
Featherless AI · Eleuther AI
Positional LSH: Binary Block Matrix Approximation for Attention with Linear Biases
May 10, 2026Daniel Wolfson, Tal WagnerSelf-Attention Long-Context Language Modeling
Blavatnik School of Computer Science and AI, Tel Aviv University
VORT: Adaptive Power-Law Memory for NLP Transformers
May 9, 2026Nabil MlaikiMemory-Augmented Language Models Transformer
Department of Mathematics and Sciences, Prince Sultan University, Riyadh, Saudi Arabia
Kaczmarz Linear Attention
May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuLong-Context Language Modeling Recurrent Transformers
School of Mathematics and Statistics Xi’an Jiaotong University Xi’an, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing, China
Self-Consolidating Language Models: Continual Knowledge Incorporation from Context
May 8, 2026Zekun Wang, Anant Gupta, Zihan Dong +1Continual Learning for LLMs Long-Context Language Modeling
Georgia Institute of Technology
Long Context Pre-Training with Lighthouse Attention
May 7, 2026Bowen Peng, Subho Ghosh, Jeffrey QuesnelleAttention Mechanisms Self-Attention
Nous Research
Matrix-Decoupled Concentration for Autoregressive Sequences: Dimension-Free Guarantees for Sparse Long-Context Rewards
May 7, 2026Pei-Sen LiLong-Context Language Modeling Autoregressive Language Modeling
School of Mathematics and Statistics, Beijing Institute of Technology, Beijing 100872, China
Training Transformers for KV Cache Compressibility
May 7, 2026Yoav Gelberg, Yam Eitan, Michael Bronstein +2Long-Context Language Modeling KV-Cache Compression
University of Oxford · 2Technion – Israel Institute of Technology · 3AITHYRA +1
Caracal: Causal Architecture via Spectral Mixing
Apr 30, 2026Bingzheng Gan, Tianyi Zhang, Yusu Li +4Long-Context Language Modeling Autoregressive Language Modeling
Huawei Technologies Co., Ltd., China.
Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
Apr 27, 2026Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh +7Long-Context Language Model Inference Long-Context Language Modeling
1AMD
Kwai Summary Attention Technical Report
Apr 27, 2026Chenglong Chu, Guorui Zhou, Guowang Zhang +35Self-Attention KV Caching
Kuaishou
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Apr 26, 2026DeepSeek-AI, Anyi Xu, Bangcai Lin +316Mixture-of-Experts Language Models Long-Context Language Model Inference
Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences
Apr 22, 2026Neehal Tumma, Noel Loo, Daniela RusLong-Context Language Modeling Recurrent Neural Networks
1MIT · 2Liquid AI
Sessa: Selective State Space Attention
Apr 20, 2026Liubomyr HorbatkoAttention Mechanisms Self-Attention
OPSDL: On-Policy Self-Distillation for Long-Context Language Models
Apr 19, 2026Xinsen Zhang, Zhenkai Ding, Tianjun Pan +4Long-Context Language Modeling On-Policy Self-Distillation
Baidu Inc.
Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation
Apr 15, 2026Zichong Li, Chen Liang, Liliang Ren +3Long-Context Language Modeling Language Model Robustness
Work is done during internship at Microsoft. · Georgia Institute of Technology · Microsoft.
SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention
Apr 15, 2026Hongtao Xu, Jianchao Tan, Yuxuan Hu +8Sparse Attention Distributed Training
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · Meituan, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Latent-Condensed Transformer for Efficient Long Context Modeling
Apr 14, 2026Zeng You, Yaofo Chen, Qiuwu Chen +5Long-Context Language Modeling LLM Inference Acceleration
South China University of Technology · Pengcheng Laboratory · AIGCode +2

May 23, 2026J/K move · Enter open · S save

Interdomain Attention: Beyond Token-Level Key-Value Memory

Naoki Kiyohara, Harrison Bo Hua Zhu, Riccardo El Hassanin, Zhuo Sun+3

Imperial College London, UK · Canon Inc., Japan · University of Copenhagen, Denmark+2

Self-Attention Long-Context Language Modeling Efficient Attention Linear Attention State Space Models

Open paper AskPDF arXiv

Softmax attention-based transformers and recurrent memory-based models (e.g., state space models (SSMs)) form the foundation of modern sequence models. Hybrid architectures aim to combine the expressivity from attention and the computational efficiency from recurrent memory, yet so far common practices simply stack both attention and recurrent layers. We propose Interdomain Attention (IA) as a novel hybrid attention architecture that mixes attention and recurrent memory in the same layer. Motivated by a kernel regression view of softmax attention, IA compresses values and kernel features of keys into a recurrent memory parameterized by an SSM. Then IA approximates dot-product attention by implicitly reconstructing the attention matrix and values from memory states; importantly this construction maintains SSM's computational efficiency advantage. We demonstrate IA's efficacy by designing and pre-training IA network architectures, where ablation studies identify attention-like readout as the primary source of gain over SSMs. Across 125M-1.3B models pre-trained on text data, IA improves over an S4D baseline at matched memory size. It also outperforms softmax attention-based Transformer in perplexity and accuracy on common-sense reasoning tasks. Based on Caduceus backbone, IA matches or outperforms the original SSM-based architecture in both perplexity and downstream tasks on genomic data.