Long-Context Language Modeling

Follow this topic Ask about this topic

Momentum

12 papers in the last four weeks, up 71% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 112

All topics

Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position
Oct 7, 2026Xiaoran Liu, Ziwei He, Xipeng QiuLong-Context Language Modeling Hybrid Attention
Fudan University · Shanghai Innovation Institute · OpenMOSS Team
Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs
Oct 5, 2026Hyunji Lee, Joykirat Singh, Zaid Khan +5Long-Context Language Modeling Recurrent Transformers
UNC Chapel Hill · The University of Texas at Austin · Mila +1
Domain adaptation of Russian ModernBERT for long legal documents
Oct 5, 2026I. Litvak, D. Gvozdetsky, F. Lashkin +7Language Model Pretraining Legal NLP
Independent Researcher · National Research Nuclear University MEPhI · Moscow Center for Advanced Studies +1
Learning to Learn a Language
Oct 5, 2026Lennart Carstens-Behrens, Holger FröhlichLanguage Model Pretraining Meta-Learning
Fraunhofer Institute for Algorithms and Scientific Computing SCAI · University Hospital Bonn
HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing
Oct 5, 2026Zhuokun Chen, Xi Lin, Xiyu Wu +3Long-Context Language Modeling Linear Attention
Monash University · Zhejiang University
RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures
Sep 30, 2026Yuyang Wu, Yufeng Du, Hao PengRotary Positional Embeddings Long-Context Language Model Evaluation
University of Illinois at Urbana-Champaign, USA
Retrieval Capacity of Self-Attention Under Competition
Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateSelf-Attention Long-Context Language Modeling
University of Luxembourg, Luxembourg
ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction
Sep 29, 2026Zheyu Shen, Guanhua Wang, Dezhan Tu +5Long-Context Language Modeling KV-Cache Management
University of Maryland, College Park · Meta
MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution
Sep 28, 2026Prasoon Dev, Anirudh Sankar, Vasudeva VarmaLong-Context Language Modeling Gated Attention
Language Technologies Research Center International Institute of Information Technology Hyderabad Hyderabad, Telangana, India
RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings
Sep 28, 2026Jarod Lévy, Mathurin Videau, Jad Yehya +3Rotary Positional Embeddings Long-Context Language Modeling
Meta AI, Paris · Inria, Université Paris-Saclay, Palaiseau, France
GSM: Efficient Language Modeling with Shared Global State
Sep 27, 2026Yunao Zheng, Bin Wen, Xiaojie Wang +12Long-Context Language Model Inference Long-Context Language Modeling
Beijing University of Posts and Telecommunications. · Kuaishou Technology.
Log-Depth Recurrent Language Modeling
Sep 23, 2026Yiqin Wang, Nuri Cingillioglu, Charles PertLong-Context Language Modeling Autoregressive Language Modeling
Department of Computing Imperial College London London, United Kingdom
Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training
Sep 16, 2026Tarun Suresh, Pranshu Chaturvedi, Hangoo Kang +4Communication-Efficient Distributed Training Long-Context Language Modeling
Stanford University
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
Sep 14, 2026Zhiwei Li, Lei Zhu, Hao Gu +6Long-Context Language Modeling Sparse Attention
Tencent HY LLM Frontier · Hong Kong University of Science and Technology (Guangzhou) · Hong Kong University of Science and Technology
Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?
Sep 8, 2026Sara Rizwan, Samaanah Abdus SalamLong-Context Language Model Evaluation Long-Context Language Modeling
Shadan Women’s College of Engineering and Technology Hyderabad, Telangana, India
Content-Based Addressing for Long Context
Sep 7, 2026Mahesh GodavartiRotary Positional Embeddings Long-Context Language Modeling
A Carrot, Inc.
CEDAR: Error-Bounded Residual Routing for Efficient Long-Context Attention
Sep 7, 2026Siyu Li, Dong Wang, Jie Zhou +3Long-Context Language Modeling Efficient Attention
School of Computer Science & Technology, Shanghai University · School of Computer Science & Technology, East China Normal University · School of Computer Engineering, Jiangsu Ocean University
BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training
Sep 2, 2026Bigyan Ghimire, Jon C. CalhounCommunication-Efficient Distributed Training Long-Context Language Modeling
Holcombe Department of Electrical and Computer Engineering Clemson University Clemson, USA
Language Models Can Control Their Own Attention
Sep 2, 2026Namgyu Ho, Huzama Ahmad, Woosung Koh +3Efficient Language Model Inference Long-Context Language Modeling
KAIST AI · Google DeepMind
Polish ModernBERT: The Long and Short of Polish Language Understanding
Sep 1, 2026Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata +1Language Model Pretraining Long-Context Language Model Evaluation
National Information Processing Institute, Warsaw, Poland
A.X K2 Technical Report
Aug 31, 2026Cheolseung Baek, Dhammiko Arya, Eunki Kim +40Mixture-of-Experts Language Models Long-Context Language Modeling
Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
Aug 12, 2026Arda Uzunoglu, Benjamin Van Durme, Daniel KhashabiLong-Context Language Modeling Factual Knowledge in Language Models
Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA
Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension
Aug 10, 2026Amanda Bertsch, Luca Soldaini, Matthew R. Gormley +4Transformer Long-Context Language Modeling
Ai2 · Carnegie Mellon University · University of Washington
Motif 3: Technical Report
Aug 10, 2026Junghwan Lim, Joon Son Chung, Sungmin Lee +24Mixture-of-Experts Language Models Long-Context Language Modeling
Motif Technologies
OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
Aug 5, 2026Indraneil Paul, Falko Helm, Goran Glavaš +1Software Engineering Long-Context Language Modeling
UKP Lab & Center for Applied Cybersecurity ATHENE, TU Darmstadt · CAIDAS, University of Würzburg
Training-Free Hashing-Based Attention via Binary Principal Components
Aug 5, 2026Daohai Yu, Zhanpeng Zeng, Keyu Chen +6Long-Context Language Modeling LLM Inference Acceleration
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Tencent YouTu Lab, Shenzhen, China · Sino-Russian Research Center for Digital Economy.
Mamba with Hierarchical Memory: Solving Representation Bottleneck in Long Sequence Modeling
Aug 3, 2026Qinwen Wang, Jieping Luo, Aoxiang Qin +5Long-Context Retrieval Memory-Augmented Neural Networks
The Chinese University of Hong Kong, Shenzhen · Hainan Bielefeld University of Applied Sciences · BrainGalaxy
DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
Aug 3, 2026Yixiao Qian, Song Chen, Pengkai Wang +3Memory-Augmented Language Models KV Caching
College of Control Science and Engineering, Zhejiang University · Department of Mathematics, National University of Singapore · Hong Kong Polytechnic University +1
Learning What to Remember: Test-Time Training via Context Distillation
Aug 3, 2026Zixuan Wang, Xingyu Dang, Rui-Jie Zhu +4Continual Learning for LLMs Long-Context Language Modeling
Princeton University · UC Santa Cruz · Carnegie Mellon University +1
Mergeable Model-Side Aggregation States for Long-Context Language Models
Jul 29, 2026Dachuan Song, Junyu Yin, Zechen Hu +1Numerical Reasoning in Language Models Long-Context Language Modeling
Department of Electrical and Computer Engineering, George Mason University, Fairfax, VA, USA · Department of Computer Science, George Mason University, Fairfax, VA, USA

Oct 7, 2026, cs.CLJ/K move · Enter open · S save

Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position

Xiaoran Liu, Ziwei He, Xipeng Qiu

Fudan University · Shanghai Innovation Institute · OpenMOSS Team

Long-Context Language Modeling Hybrid Attention Linear Attention Positional Encoding

Open paper AskPDF arXiv

The architectural design of Large Language Models (LLMs) is shifting from traditional full-attention-only models to hybrid models, which combine different attention modules to improve long-context efficiency and performance in length extrapolation and context extension. To explain why hybrid models work and how to design them better, we propose Mechanics of Long-Context Hybrid Models. As Part 1.1 of this series, we begin with hybrids of full attention and either sliding-window attention (SWA) or gated variants of linear attention (LA), represented by GLA and GDN. We first observe a Seesaw Effect in Context Extension: LA hybrids benefit more from long-context continual pretraining, whereas SWA hybrids perform better under length extrapolation. We attribute this behavior to differences in the positional inductive biases induced by these attention mechanisms. We find that SWA hybrids suffer from a Short-Context Learning Trap, Short-Window Weariness, and Long-Window Laziness, and require extended windows to enhance performance in continual long-context pretraining. For LA hybrids, we summarize the Matthew Effect of Hybrid Position Extrapolation and propose Sliding-Window Linear Attention, achieving 16

\times

training-free length extrapolation while maintaining 100% accuracy on NIAH-SK1 in 64k context length.

Momentum

Related topics

Latest papers 112