Long-Context Language Modeling

Follow this topic Ask about this topic

The field asks how models can retrieve relevant evidence and maintain useful context as sequences grow, for tasks such as document understanding, code analysis, and long-range recall. Approaches include sparse and hierarchical attention, positional-encoding extensions, recurrent and state-space memory, hybrid architectures, and compression of context or key-value caches. Training methods range from long-sequence continued pretraining and curricula to distillation and test-time adaptation, with evaluation often using retrieval and reasoning benchmarks such as RULER, LongBench, and Needle-in-a-Haystack. The central tension is reducing compute and memory without losing reliable access to distant information; performance also depends on information density and may not extrapolate from short training contexts.

Latest papers

189 papers38 papers in the last four weeks, up 90% on the four weeks before. 0.2% of all new papers.

SinkTrack: Attention Sink based Context Anchoring for Large Language Models
Apr 11, 2026Xu Liu, Guikun Chen, Wenguan WangLLM Context Management LLM Hallucination Mitigation
The State Key Lab of Brain-Machine Intelligence, Zhejiang University
PolicyLong: Towards On-Policy Context Extension
Apr 9, 2026Junlong Jia, Jiang Zhou, Ziyang Chen +5Long-Context Language Modeling Self-Training for Language Models
Hunyuan Team, Tencent · Chinese Academy of Sciences
Optimal Decay Spectra for Linear Recurrences
Apr 8, 2026Yang CaoLinear RNNs Long-Range Dependency Modeling
Short Data, Long Context: Distilling Positional Knowledge in Transformers
Apr 7, 2026Patrick Huber, Ernie Chang, Chinnadhurai Sankar +4Long-Context Language Modeling LLM Context Management
Meta Reality Labs
Residual-Mass Accounting for Partial-KV Decoding
Apr 7, 2026Yasuto Hoshi, Daisuke Miyashita, Jun DeguchiSoftmax Attention Long-Context Language Modeling
Kioxia Corporation
CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling
Apr 5, 2026Dejan Čugalj, Aleksandar JevremovicLong-Context Language Modeling Sequence Modeling
Singidunum University
Screening Is Enough
Apr 1, 2026Ken M. NakanishiSelf-Attention Long-Context Language Modeling
Center for Emergent Matter Science (CEMS), RIKEN · Graduate School of Science, The University of Tokyo
Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers
Mar 27, 2026Yusheng Zhao, Hourun Li, Bohan Wu +5Hybrid Transformer Architectures Hybrid Attention
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University, Beijing, China · Contribution during internship at Huawei Technologies Co., Ltd. · Huawei Technologies Co., Ltd.
Density-aware Soft Context Compression with Semi-Dynamic Compression Ratio
Mar 26, 2026Yijiong Yu, Shuai Yuan, Jie Zheng +2LLM Context Compression Context Compression
Oregon State University · DeepSolution
Large Language Model as Token Compressor and Decompressor
Mar 26, 2026Wenbing Li, Yiran Wang, Zikai Song +4LLM Context Compression LLM Compression
MemDLM: Memory-Enhanced DLM Training
Mar 23, 2026Zehua Pei, Hui-Ling Zhen, Weizhe Lin +4Diffusion Language Model Training Diffusion Language Models
The Chinese University of Hong Kong · Huawei Technologies Co., Ltd
TAMTRL: Teacher-Aligned Reward Reshaping for Multi-Turn Reinforcement Learning in Long-Context Compression
Mar 23, 2026Li Wang, Yandong Wang, Xin Yu +3RL for Language Models Long-Context Language Modeling
School of Artificial Intelligence, Beihang University, Beijing, 100191, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China · Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University, Beijing, China
HiCI: Hierarchical Construction-Integration for Long-Context Attention
Mar 21, 2026Xiangyu Zeng, Qi Xu, Yunke Wang +1Long-Context Language Modeling Long-Context Modeling
School of Computer Science, University of Sydney, Sydney, Australia · 2Westlake University, Hangzhou, China.
Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression
Mar 21, 2026Ruijie Miao, Zhiming Wang, Wang Li +4KV-Cache Management KV-Cache Compression
Peking University · ByteDance Seed
MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning
Mar 21, 2026Dong Liu, Yanxuan Yu, Ben Lengerich +1Attention Mechanisms Memory-Efficient Attention
University of California, Los Angeles, Los Angeles, California, USA · Columbia University, New York, New York, USA · University of Wisconsin-Madison, Madison, Wisconsin, USA
Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation
Mar 20, 2026Yehjin Shin, Seojin Kim, Noseong ParkSelective SSMs Long-Context Language Modeling
KAIST
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Mar 18, 2026Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4Self-Attention Long-Context Language Modeling
Department of Electrical and Computer Engineering, Purdue University, USA · AWS Agentic AI, USA
M $^2$ RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
Mar 15, 2026Mayank Mishra, Shawn Tan, Ion Stoica +2Recurrent Neural Networks Language Modeling
UC Berkeley · Learning Machine · MIT-IBM Watson Lab +2
LPC-SM: Local Predictive Coding and Sparse Memory for Long-Context Language Modeling
Mar 12, 2026Keqin XieLong-Context Language Modeling Autoregressive Language Modeling
Independent Researcher, Suzhou, China
Long-Context Encoder Models for Polish Language Understanding
Mar 12, 2026Sławomir Dadas, Rafał Poświata, Marek Kozłowski +4Long-Context Language Modeling Long-Context Modeling
National Information Processing Institute, Warsaw, Poland · PKO Bank Polski, Warsaw, Poland
LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
Mar 9, 2026Ning Yang, Hengyu Zhong, Wentao Wang +4Attention-Guided Distillation Memory-Efficient Attention
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Southwest University, Chongqing, China · Dalian University of Technology, Dalian, Liaoning, China +3
MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens
Mar 6, 2026Yu Chen, Runkai Chen, Sheng Yi +9Memory-Augmented Language Models Long-Context Language Modeling
Evermind · Shanda Group · Peking University
Stacked from One: Multi-Scale Self-Injection for Context Window Extension
Mar 5, 2026Wei Han, Pan Zhou, Soujanya Poria +1Long-Context Language Model Inference Long-Context Language Modeling
Singapore University of Technology and Design (SUTD), Singapore · Singapore Management University (SMU), Singapore · Nanyang Technological University (NTU), Singapore +1
MaBERT:A Padding Safe Interleaved Transformer Mamba Hybrid Encoder for Efficient Extended Context Masked Language Modeling
Mar 3, 2026Jinwoong Kim, Sangjin ParkHybrid Transformer-State Space Model Architectures Long-Context Language Modeling
Graduate School of Industrial Data Engineering, Hanyang University, Seoul, Republic of Korea
Understanding the Physics of Key-Value Cache Compression for LLMs through Attention Dynamics
Mar 2, 2026Samhruth Ananthanarayanan, Ayan Sengupta, Tanmoy ChakrabortyAttention Routing KV-Cache Compression
IIT Delhi, India
Memory Caching: RNNs with Growing Memory
Feb 27, 2026Ali Behrouz, Zeman Li, Yuan Deng +3Recurrent Neural Networks Memory-Augmented Language Models
Frayed RoPE and Long Inputs: A Geometric Perspective
Feb 24, 2026Davis Wertheimer, Aozhong Zhang, Derrick Liu +2Rotary Positional Embeddings Long-Context Language Modeling
IBM Research · University at Albany, SUNY
Whole-Piece Training for Symbolic Music Language Models via Full-Horizon Compressed Recurrence
Feb 23, 2026Yungang Yi, Weihua Li, Matthew Kuo +2Long-Context Language Modeling Symbolic Music Generation
Auckland University of Technology, Auckland, New Zealand · University of Tasmania, Tasmania, Australia
Reinforced Fast Weights via Next-Sequence Prediction
Feb 18, 2026Hee Seung Hwang, Xindi Wu, Sanghyuk Chun +2Fast Weights Long-Context Language Modeling
Princeton University
Fast KV Compaction via Attention Matching
Feb 18, 2026Adam Zweiger, Xinghong Fu, Han Guo +1KV-Cache Management LLM Context Compression
Massachusetts Institute of Technology.

Apr 11, 2026J/K move · Enter open · S save

SinkTrack: Attention Sink based Context Anchoring for Large Language Models

Xu Liu, Guikun Chen, Wenguan Wang

The State Key Lab of Brain-Machine Intelligence, Zhejiang University

LLM Context Management LLM Hallucination Mitigation Long-Context Language Modeling

Open paper AskPDF arXiv

Large language models (LLMs) suffer from hallucination and context forgetting. Prior studies suggest that attention drift is a primary cause of these problems, where LLMs' focus shifts towards newly generated tokens and away from the initial input context. To counteract this, we make use of a related, intrinsic characteristic of LLMs: attention sink -- the tendency to consistently allocate high attention to the very first token (i.e., <BOS>) of a sequence. Concretely, we propose an advanced context anchoring method, SinkTrack, which treats <BOS> as an information anchor and injects key contextual features (such as those derived from the input image or instruction) into its representation. As such, LLM remains anchored to the initial input context throughout the entire generation process. SinkTrack is training-free, plug-and-play, and introduces negligible inference overhead. Experiments demonstrate that SinkTrack mitigates hallucination and context forgetting across both textual (e.g., +21.6% on SQuAD2.0 with Llama3.1-8B-Instruct) and multi-modal (e.g., +22.8% on M3CoT with Qwen2.5-VL-7B-Instruct) tasks. Its consistent gains across different architectures and scales underscore the robustness and generalizability. We also analyze its underlying working mechanism from the perspective of information delivery. Our source code is available at https://github.com/67L1/SinkTrack.