Long-Context Modeling

Follow this topic Ask about this topic

Momentum

11 papers in the last four weeks, up 83% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 105

All topics

Context by Distinct Information: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams
Jul 11, 2026Siddharth Pal, Viktoria RojkovaMemory-Augmented Language Models Long-Context Modeling
SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding
Jul 11, 2026Abhigya Verma, Khyati Mahajan, Amit Kumar Saha +4VLM Evaluation Vision-Language Models
ServiceNow AI · Mila · Université de Montréal
DSSMs: State Space Models with Explicit Memory via Delay Differential Equations
Jul 11, 2026Yixiao Qian, Song Chen, Jiaxu Liu +2Dynamical Systems Long-Context Modeling
College of Control Science and Engineering, Zhejiang University · Department of Mathematics, National University of Singapore · School of Mathematical Sciences, Zhejiang University
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Jul 3, 2026Xiang Hu, Xinyu Wei, Hao Gu +10Long-Context Language Model Inference Long-Context Language Modeling
Tencent HY Team · ShanghaiTech University · The Hong Kong University of Science and Technology +1
PraMem: Practice-derived Experiential Memory for Long-horizon Behavior Prediction
Jul 3, 2026Zhuoqun Li, Boxi Cao, Jiawei Chen +11Memory-Augmented Language Models Next-Action Prediction
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Fudan University +1
Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale
Jul 1, 2026Siddharth Gollapudi, Nilesh Gupta, Prasann Singhal +1In-Context Retrieval Length Generalization
UC Berkeley · UT Austin
Multi-Head Recurrent Memory Agents
Jul 1, 2026Jiatong Li, Samuel Yeh, Sharon LiLLM Memory Long-Context Modeling
Department of Computer Science, University of Wisconsin-Madison
Hierarchical Global Attention (HGA)
Jun 29, 2026Woernle Frank, Fedosov Vladimir, Grinenko ArtemiyKV-Cache Offloading Long-Context Modeling
BMW Group
MuonSSM: Orthogonalizing State Space Models for Sequence Modeling
Jun 29, 2026Thai-Khanh Nguyen, Ngoc-Bich-Uyen Vo, Thieu N. Vo +2Long-Context Modeling Sequence Modeling
Faculty of Information Technology, Dainam University, Hanoi University of Science and Technology, Hanoi 10000, Vietnam · Faculty of Artificial Intelligence, Posts and Telecommunications Institute of Technology, Hanoi 10000, Vietnam · Department of Computer Science, University of Bath, Bath BA2 7AY, UK +1
Diagnosing and Mitigating Context Rot in Long-horizon Search
Jun 29, 2026Shijie Xia, Yikun Wang, Zhen Huang +1Long-Context Modeling LLM Context Management
1Shanghai Jiao Tong University · 3SII · 4GAIR +1
The Verbose Context Problem in Medical Records
Jun 28, 2026Shiva Kaul, Min-Gyu Kim, Anjum Khurshid +1Long-Context Modeling Longitudinal EHR Modeling
Department of Population Medicine · Department of Biomedical Informatics, Ajou University School of Medicine · Department of Population Medicine, Harvard Pilgrim Health Care Institute and Harvard Medical School +1
How Token Influence Decays with Distance: A Green-Function View of Trained Language Models
Jun 28, 2026Matthias Brändel, Stephan Köhler, Oliver RheinbachTransformer Interpretability Autoregressive Language Modeling
Faculty of Mathematics and Computer Science Technische Universität Bergakademie Freiberg, Freiberg, Germany
Depth-Staggered Fibonacci Spacing for Sparse Attention: Static Schedules Beat Learned Dilation and Extrapolate Where Dense Attention Fails
Jun 26, 2026Chad A. CappsStructured Sparsity Long-Context Modeling
Independent Researcher
Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention
Jun 24, 2026Luke McDermott, Robert W. Heath, Rahul ParhiLifelong Learning Agents Long-Context Modeling
UC San Diego
Why Do Accumulated Transformations Extrapolate?
Jun 23, 2026Mahesh GodavartiRotary Positional Embeddings Length Generalization
A Carrot, Inc.
Scaling State-Space Models from Lines to Paragraphs: An Ablation of Mamba-based OCR
Jun 22, 2026Merveilles Agbeti-Messan, Pierrick Tranouez, Stéphane Nicolas +2Selective SSMs Long-Context Modeling
LITIS EA4108, University of Rouen Normandy, France · LITIS EA4108, INSA of Rouen Normandy, France
moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT
Jun 21, 2026Thiago Laitz, Thales Sales Almeida, João Guilherme Alves Santos +1Language Model Pretraining Long-Context Modeling
UNICAMP, Campinas, Brazil · Tropic AI · Maritaca AI
HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning
Jun 19, 2026Awais Rauf, Ahmed Hasssan, Greg SlabaughVideo Understanding Long-Context Modeling
Queen’s University Belfast · United Kingdom · AMD +2
HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization
Jun 18, 2026Zhentao Tan, Wei Chen, Jingyi Shen +4Softmax Attention Long-Context Modeling
Alibaba Group
Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
Jun 17, 2026Xiaoyue Xu, Sikui Zhang, Xiaorong Wang +2Training Data Curation RL for Language Model Reasoning
OpenBMB · Tsinghua University
How Well Can Your Video Model Remember? Measuring Memory-Budget Trade-offs in Long Video Understanding
Jun 17, 2026Yixian TianVLM Evaluation Streaming Video Understanding
1Fudan University · 2Beijing University of Posts and Telecommunications
Training and Evaluating Diffusion Policies with Long Context Lengths
Jun 15, 2026Abhinav Agarwal, Adam Wei, Taylan Kargin +6Diffusion Policy Long-Context Modeling
Massachusetts Institute of Technology
Rethinking the Role of Efficient Attention in Hybrid Architectures
Jun 13, 2026Ziqing Qiao, Yinuo Xu, Chaojun Xiao +6Attention Head Analysis Long-Context Modeling
1Tsinghua University · 2OpenBMB
LongSpike: Fractional Order Spiking State Space Models for Efficient Long Sequence Learning
Jun 11, 2026Xinrui He, Qiyu Kang, Xuhao Li +1Long-Context Modeling Spiking Neural Networks
Wuhan University · University of Science and Technology of China · Anhui University
Context-Driven Incremental Compression for Multi-Turn Dialogue Generation
Jun 10, 2026Yeongseo Jung, Jaehyeok Kim, Eunseo Jung +5Memory-Augmented Language Models Long-Context Modeling
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong · NVIDIA AI Technology Center, NVIDIA · University of Tsukuba, Japan +1
Attention Expansion: Enhancing Keyphrase Extraction from Long Documents with Attention-Augmented Contextualized Embeddings
Jun 9, 2026Roberto Martínez-Cruz, Alvaro J. López-López, José PortelaDocument Information Extraction Long-Context Modeling
Institute for Research in Technology, ICAI School of Engineering, Comillas Pontifical University, Madrid, Spain. · DD-AIM, New York, USA.
Kwai Keye-VL-2.0 Technical Report
Jun 9, 2026Kwai Keye Team, Bin Wen, Changyi Liu +50Temporal Video Grounding Long-Context Modeling
Customer-Agent: Overcoming Context Limitations in Ultra-Long Shopping Trajectories via Tool-Augmented Agents and RLVR
Jun 6, 2026Hongye Liu, Rongmei Lin, Anurag Kashyap +4Tool-Augmented Language Model Agents Long-Context Modeling
Amazon · Duke University
MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios
Jun 5, 2026Haitian Wang, Ruoxi Sun, Quantong Qiu +5Multimodal IR Multimodal Embedding
Soochow University, Suzhou, China · Ant Group, Beijing, China
EASE-TTT: Evidence-Aligned Selective Test-Time Training for Long-Context Question Answering
Jun 5, 2026Xiaopeng Yuan, Zebin Wang, Suwen Wang +3Long-Context Retrieval Long-Context QA
University of Illinois Urbana-Champaign · 2Harvard University · 3Brion, ASML US LP +1

Jul 11, 2026, cs.LGJ/K move · Enter open · S save

Context by Distinct Information: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams

Siddharth Pal, Viktoria Rojkova

Memory-Augmented Language Models Long-Context Modeling LLM Context Management

Open paper AskPDF arXiv

Context engineering decides what information a model carries forward, and current designs meter it in tokens: compressing the past into a bounded recurrent state, keeping a key-value entry for every token, or imposing a fixed budget through a window or eviction rule. All three make the token the unit of memory even when the stream is redundant and the task depends on the distinct information it carries. Building on a companion mechanism paper that opens a cache slot only when an incoming key is novel, so memory scales with the number of distinct items rather than tokens, we develop that allocate-on-novelty cache as a working-memory component and organize context by how a task depends on the past: recall-carried information belongs in a content-addressed novelty cache, summary-carried information in a recurrent state, and locality-carried information in a recency window. The claim is empirical and bounded. On a matched character-level control, novelty-gated attention reaches full-attention performance while attending to about half the tokens, and coupling the cache with a state-space summary matches full-attention coupling at that reduced cost; the advantage grows as context lengthens, while a sliding window is preferable on short, locality-dominated spans. On next-code prediction over synthetic Medicare claims the coupled component leads full attention and every fixed-budget eviction policy at a thousand-event horizon, whereas cost forecasting over the same stream is summary-carried and the cache is neutral. The retained memory is an inspectable table of templates, codes, drugs, or places rather than an opaque state. The experiments are small-scale and use only public data; they establish the primitive that context can scale with distinct information rather than tokens, in a working memory that is content-addressable and auditable.

Momentum

Related topics

Latest papers 105