Long-Context Language Modeling

Ask about this topic

Overview

The field asks how models can retrieve relevant evidence and maintain useful context as sequences grow, for tasks such as document understanding, code analysis, and long-range recall. Approaches include sparse and hierarchical attention, positional-encoding extensions, recurrent and state-space memory, hybrid architectures, and compression of context or key-value caches. Training methods range from long-sequence continued pretraining and curricula to distillation and test-time adaptation, with evaluation often using retrieval and reasoning benchmarks such as RULER, LongBench, and Needle-in-a-Haystack. The central tension is reducing compute and memory without losing reliable access to distant information; performance also depends on information density and may not extrapolate from short training contexts.

Latest papers

CardsList

189 papers38 papers in the last four weeks, up 90% on the four weeks before. 0.2% of all new papers.

  1. Multi-State Linear Attention with Generalized Second-Order Routing

    Oct 27, 2026Hee Seung Hwang, Sanghyuk Chun, Xindi Wu +2Long-Context Language ModelingLinear Attention

    Princeton University · Google DeepMind

  2. Head Dimension Matters in Language Models for Long-Context Retrieval

    Oct 23, 2026Seiya Tokui, Shoichiro Yamaguchi, Sosuke KobayashiLong-Context RetrievalAttention Head Analysis

    Preferred Networks, Inc., Tokyo, Japan

  3. HyScale: Scaling Long-Context Training for Learned Sparse Attention

    Oct 23, 2026Yangshen Deng, Xinjing Huang, Yulong Wang +8Efficient Language Model TrainingSparse Attention

    University of Edinburgh · Tencent Hunyuan

  4. LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning

    Oct 22, 2026Keqi Deng, Ruchao Fan, Jinyu LiSparse Fine-TuningLLM Fine-Tuning

    Microsoft, USA

  5. Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

    Oct 22, 2026Aman Behera, Namit Solanki, Mehul AnandState Space Language ModelsLong-Context Language Modeling

    IIT Roorkee · AISSMS IOIT, Pune · Independent Researcher

  6. OpenAttention: One Percent Is Enough with Tail Proxies

    Oct 22, 2026Hyunwoo Yu, Jongbeom Lee, Hyunbin Kim +8Efficient AttentionSparse Attention

    Sogang University · Samsung Electronics · Pusan National University

  7. Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law

    Oct 21, 2026Ziqing Qiao, Xiaoyue Xu, Sikui Zhang +4Long-Context Language ModelingContinued Pretraining

    Department of Computer Science and Technology, Tsinghua University, Beijing, China · OpenBMB

  8. Kronecker Delta Rule for Long Context Linear Attention

    Oct 19, 2026Anamaria-Roberta Hartl, Levente Zólyomi, Michael List +3Delta-Rule Linear AttentionAttention Mechanisms

    Institute for Machine Learning, Johannes Kepler University, Linz · NXAI GmbH, Linz

  9. Which Long Documents Matter? A Controlled Study of Data Selection for Context Extension

    Oct 17, 2026Junlong Jia, Jiang Zhou, Xing Wu +3Training Data SelectionLong-Context Language Modeling

    Hunyuan Team, Tencent

  10. First Passage Attention

    Oct 15, 2026Yuxuan LiuLong-Context Language ModelingSoftmax Attention

    Department of Mathematics, University of Bergen, Bergen, Norway

  11. Copy-Paste Language Models

    Oct 13, 2026Nathan Godey, Yoav ArtziLanguage Model PretrainingEfficient Language Model Training

    Cornell University New York City, USA

  12. ChampionKV: Head-Wise Adaptive Budget Allocation for KV Cache Compression

    Oct 13, 2026Yuechuan Chen, Yanda Guo, Xiaoxi Liu +4KV-Cache CompressionKV-Cache Management

    Sun Yat-sen University · The Chinese University of Hong Kong · University of Electronic Science and Technology of China +2

  13. Virtual Head Attention

    Oct 13, 2026Xibo Ding, Guoxia Wang, Jinle Zeng +4Long-Context Language ModelingGrouped-Query Attention

    Baidu Inc. · Peking University

  14. FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation

    Oct 12, 2026Zekun Shi, Alastair James Arthur PriceLong-Context Language ModelingLong-Context Modeling

    National University of Singapore · Sea AI Lab, Singapore · University of Toronto

  15. FON: A Draft-Revise Dual-Organ Language Model without Sequence-Level Quadratic Attention

    Oct 12, 2026Helong Hu, Hongdan Pan, Shuiqing HuLanguage ModelingSequence Modeling

    Zhanjiang Campus, Guangdong Ocean University, Zhanjiang, Guangdong, China · Maoming Administrative Service Center, Maoming, Guangdong, China · China Pacific Insurance

  16. Apertus 1.5: From Open Text Models to Open Multimodal Reasoners

    Oct 11, 2026Project Apertus, :, Yixuan Xu +130MLLM ReasoningMultimodal Large Language Models

    ETH Zurich · EPFL · HES-SO Valais-Wallis +5

  17. Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers

    Oct 9, 2026Yuhong Chou, Zehao Liu, Yuqi Pan +4Transformer AttentionAttention Mechanisms

    The Hong Kong Polytechnic University · University of Chinese Academy of Sciences · Independent Researcher +1

  18. RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts

    Oct 8, 2026Frederikke Isa Marin, Panagiotis Antoniadis, Dionysia Danai Brilli +5GenomicsBioinformatics

    University of Copenhagen · Novo Nordisk A/S · Technical University of Denmark

  19. BioBigBird: A Sparse Attention Model for Long-Range Dependency Processing in Biomedical Text

    Oct 8, 2026Roshan Balaji, Pavan Kumar S, Vasudev Gupta +3Language ModelingLong-Context Language Modeling

    Wadhwani School of Data Science and AI, Indian Institute of Technology Madras, Chennai, India

  20. Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position

    Oct 7, 2026Xiaoran Liu, Ziwei He, Xipeng QiuLong-Context Language ModelingHybrid Attention

    Fudan University · Shanghai Innovation Institute · OpenMOSS Team

  21. Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs

    Oct 5, 2026Hyunji Lee, Joykirat Singh, Zaid Khan +5Long-Context Language ModelingRecurrent Transformers

    UNC Chapel Hill · The University of Texas at Austin · Mila +1

  22. Domain adaptation of Russian ModernBERT for long legal documents

    Oct 5, 2026I. Litvak, D. Gvozdetsky, F. Lashkin +7Language Model PretrainingLegal NLP

    Independent Researcher · National Research Nuclear University MEPhI · Moscow Center for Advanced Studies +1

  23. Learning to Learn a Language

    Oct 5, 2026Lennart Carstens-Behrens, Holger FröhlichLanguage Model PretrainingMeta-Learning

    Fraunhofer Institute for Algorithms and Scientific Computing SCAI · University Hospital Bonn

  24. HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

    Oct 5, 2026Zhuokun Chen, Xi Lin, Xiyu Wu +3Long-Context Language ModelingLinear Attention

    Monash University · Zhejiang University

  25. TrajLong: Co-Designing Agentic and Long-Context Supervision for Mid-Training

    Oct 4, 2026Miao Peng, Qintong Zhang, Nuo Chen +8LLM Agent TrainingLong-Context Language Modeling

    The Hong Kong University of Science and Technology (Guangzhou) · Hunyuan Team, Tencent · Peking University

  26. WavePrune: One period is often enough for RoPE

    Oct 3, 2026Guancheng Du, Luotian Huang, Shaowen Wang +2Rotary Positional EmbeddingsPositional Encoding

    Tsinghua University

  27. Length Generalization Needs Proper Regularization

    Oct 3, 2026Pavlo Vasylenko, Matthias Lindemann, André F. T. Martins +1Length GeneralizationLanguage Model Generalization

    Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · ELLIS Unit Lisbon +1

  28. SlimKV: Joint Token-Feature KV Cache Compression with Reconstruction-Free Beacon Attention

    Oct 2, 2026Zihan Teng, Jiayu Zhao, Wentao Ren +4KV-Cache CompressionLLM Inference Acceleration

    University of Science and Technology of China · Nanyang Technological University · Tencent

  29. Trained Agentic Context Management

    Oct 1, 2026Bryce SandlundLong-Context Language ModelingLong-Context Reasoning in Language Models

  30. RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures

    Sep 30, 2026Yuyang Wu, Yufeng Du, Hao PengRotary Positional EmbeddingsLong-Context Language Model Evaluation

    University of Illinois at Urbana-Champaign, USA