Long-Context Language Modeling

Overview

The field asks how models can retrieve relevant evidence and maintain useful context as sequences grow, for tasks such as document understanding, code analysis, and long-range recall. Approaches include sparse and hierarchical attention, positional-encoding extensions, recurrent and state-space memory, hybrid architectures, and compression of context or key-value caches. Training methods range from long-sequence continued pretraining and curricula to distillation and test-time adaptation, with evaluation often using retrieval and reasoning benchmarks such as RULER, LongBench, and Needle-in-a-Haystack. The central tension is reducing compute and memory without losing reliable access to distant information; performance also depends on information density and may not extrapolate from short training contexts.

Latest papers

CardsList

205 papers22 papers in the last four weeks, down 44% on the four weeks before. 0.2% of all new papers.

  1. SMELL: Sparsity-aware Memory Efficient Federated Fine-tuning for Long-context LLMs

    Nov 14, 2026Yin Xu, Yongbo Yang, He Sun +2Federated Fine-TuningMemory-Efficient Fine-Tuning

  2. Bilateral Log-Linear Attention

    Nov 13, 2026Kazi Ashik Islam, Guojing CongMemory-Efficient AttentionLong-Context Language Modeling

  3. Mask-Pattern Position Augmentation: Training-Free Long-Context Extrapolation for Diffusion LLMs

    Nov 11, 2026Gaurav Sarkar, Syed Affan DaimiLong-Context Language ModelingDiffusion Language Models

  4. Looping Deeper Without Growing the KV Cache: Recurrent Depth for Causal Encoder-Decoders

    Nov 9, 2026Jiaqi Leng, Minshen Zhang, Zhaofeng Sun +3LLM Inference AccelerationKV-Cache Management

  5. ASCENT: Scale-Complementary External State for Frozen Long-Context Language Models

    Nov 6, 2026Xingjian ZhangMemory-Augmented Language ModelsLong-Context Language Modeling

  6. Attend at the Right Resolution: Head-Adaptive Sparse Attention via Query-Key Interaction Structure

    Nov 6, 2026Wajih Hassan Raza, Ahsan Bilal, Yidan Shen +3Sparse AttentionEfficient Attention

  7. Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout

    Nov 5, 2026Tushaar Gangavarapu, Gil Tabak, Clemens JS SchaeferDelta-Rule Linear AttentionSequence Modeling

  8. Recursive Sparse Attention: Million-Token Length Extrapolation at Logarithmic Cost

    Nov 4, 2026Ruitian Wang, Xinyu Yang, Minshen Zhang +1Sparse AttentionLong-Context Language Modeling

  9. Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

    Oct 30, 2026Ales Kubicek, Andrea Caciolai, Liangzu Peng +7Sparse AttentionLong-Context Language Modeling

  10. Subquadratic Attention with a Hierarchical Tree-Structured Indexer

    Oct 30, 2026Leon Lufkin, Cutter Dawes, Alexander Speicher +3Efficient AttentionLong-Context Language Modeling

  11. RangeSWA: Training-Free Length Extrapolation for Continuous Diffusion Language Models

    Oct 30, 2026Guangfu Guo, Xiaoqian Lu, Linsey Pang +4Long-Context Language ModelingDiffusion Language Model Inference

  12. Multi-State Linear Attention with Generalized Second-Order Routing

    Oct 27, 2026Hee Seung Hwang, Sanghyuk Chun, Xindi Wu +2Long-Context Language ModelingLinear Attention

  13. Head Dimension Matters in Language Models for Long-Context Retrieval

    Oct 23, 2026Seiya Tokui, Shoichiro Yamaguchi, Sosuke KobayashiLong-Context RetrievalAttention Head Analysis

  14. HyScale: Scaling Long-Context Training for Learned Sparse Attention

    Oct 23, 2026Yangshen Deng, Xinjing Huang, Yulong Wang +8Efficient Language Model TrainingSparse Attention

  15. Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

    Oct 22, 2026Aman Behera, Namit Solanki, Mehul AnandState Space Language ModelsLong-Context Language Modeling

  16. OpenAttention: One Percent Is Enough with Tail Proxies

    Oct 22, 2026Hyunwoo Yu, Jongbeom Lee, Hyunbin Kim +8Efficient AttentionSparse Attention

  17. Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law

    Oct 21, 2026Ziqing Qiao, Xiaoyue Xu, Sikui Zhang +4Long-Context Language ModelingContinued Pretraining

  18. Kronecker Delta Rule for Long Context Linear Attention

    Oct 19, 2026Anamaria-Roberta Hartl, Levente Zólyomi, Michael List +3Delta-Rule Linear AttentionAttention Mechanisms

  19. First Passage Attention

    Oct 15, 2026Yuxuan LiuLong-Context Language ModelingSoftmax Attention

  20. Copy-Paste Language Models

    Oct 13, 2026Nathan Godey, Yoav ArtziLanguage Model PretrainingEfficient Language Model Training

  21. ChampionKV: Head-Wise Adaptive Budget Allocation for KV Cache Compression

    Oct 13, 2026Yuechuan Chen, Yanda Guo, Xiaoxi Liu +4KV-Cache CompressionKV-Cache Management

  22. Virtual Head Attention

    Oct 13, 2026Xibo Ding, Guoxia Wang, Jinle Zeng +4Long-Context Language ModelingGrouped-Query Attention

  23. FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation

    Oct 12, 2026Zekun Shi, Alastair James Arthur PriceLong-Context Language ModelingLong-Context Modeling

  24. FON: A Draft-Revise Dual-Organ Language Model without Sequence-Level Quadratic Attention

    Oct 12, 2026Helong Hu, Hongdan Pan, Shuiqing HuLanguage ModelingSequence Modeling

  25. Apertus 1.5: From Open Text Models to Open Multimodal Reasoners

    Oct 11, 2026Project Apertus, :, Yixuan Xu +130MLLM ReasoningMultimodal Large Language Models