Long-Context Modeling

Follow this topic Ask about this topic

Long-context modeling aims to let models use information spread across lengthy text, conversations, documents, videos, and time series without losing relevant details. Approaches include sparse or hierarchical attention, recurrent state-space models, external and compressed memory, retrieval, and training methods that extend context or teach evidence use. Researchers evaluate these systems on tasks such as long-document question answering, multi-hop retrieval, dialogue memory, and long-video understanding, while also studying positional encodings and how performance changes with context length. A central tension is that larger windows do not guarantee reliable reasoning: models may miss distant evidence or be distracted by irrelevant material, while efficient methods must balance computation and memory against recall and accuracy.

Latest papers

139 papers18 papers in the last four weeks, up 80% on the four weeks before. 0.1% of all new papers.

Topology Survives Shuffling: Robust Long-Context Memory Access Prediction for Graph Analytics
Oct 29, 2026Dongyan Sun, Neelesh Gupta, Rajgopal Kannan +1Long-Context Modeling Distribution Shift Robustness
University of Southern California, USA · DEVCOM Army Research Office, USA
Unlimited OCR Works 2
Oct 29, 2026Shu Wei, Jingjing Wu, Qunyi Xie +9Optical Character Recognition Long-Context Modeling
Grounding Long Financial Text with Semantically-Enriched Tables via Multimodal Learning
Oct 27, 2026Xiaoyu HuMultimodal Transformer Tabular Representation Learning
Rutgers University
GTFormer: A Syntax-Free Graph Network Alternative to Transformers for 8192 $\times$ Length Extrapolation in Linear Time
Oct 27, 2026Elisabeth FinkLong-Context Modeling Sequence Modeling
Middlesex University, Department of Computer Science The Burroughs, Hendon, London NW4 4BT, UK.
Write What Matters: Disentangling Token Selection from Slot Allocation in Recurrent Memory
Oct 25, 2026Chen Chen, Rui ChengMemory-Augmented Neural Networks Long-Context Modeling
Hong Kong Baptist University · Anhui University
Why Full-Duplex Speech Models Collapse in Long Conversations
Oct 23, 2026Yeonju Kim, Bella Godiva, Se Jin Park +1Full-Duplex Spoken Dialogue Systems Long-Context Modeling
Integrated Vision and Language Lab, KAIST, South Korea · Kyung Hee University, South Korea
Weighted State Souping for Efficient Long-Context Retrieval
Oct 19, 2026William L. Tong, Aryo Lotfi, Emmanuel Abbe +3State Space Language Models Long-Context Retrieval
Harvard University · Apple
Joint Subspace Sparse Attention for Efficient Long-Context Decoding
Oct 13, 2026Jiwon Song, Yoongon Kim, Jae-Joon KimLLM Inference Acceleration Sparse Attention
Seoul National University
FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation
Oct 12, 2026Zekun Shi, Alastair James Arthur PriceLong-Context Language Modeling Long-Context Modeling
National University of Singapore · Sea AI Lab, Singapore · University of Toronto
History Replays Itself: Unleashing Long-Context Capability in Linear Attention
Oct 11, 2026Li Nanbo, Firas Laakom, Yilan Zhang +10Long-Context Modeling Memory-Augmented Language Models
King Abdullah University of Science and Technology (KAUST)
In-Context Learning for System Identification: Probabilistic Formulation, Long-Sequence Handling, and Benchmark Validation
Oct 9, 2026Matteo Rufolo, Dario Piga, Filippo Pura +1System Identification In-Context Learning
SUPSI, IDSIA - Dalle Molle Institute for Artificial Intelligence, Lugano, Switzerland · USI, Faculty of Informatics, Lugano, Switzerland
Understanding Mamba-3 Through Oscillatory Dynamics
Oct 9, 2026Shutian Liang, Yue SongSequence Modeling Selective SSMs
College of AI, Tsinghua University
Double-Take: Improving Capacity, Binding, and Selectivity In Long-Context Within Bidirectional Memory
Oct 9, 2026Vakul Raghavan, Utkarsh Grover, Mingyang Mao +2Long-Context Modeling Memory-Augmented Neural Networks
Department of Mathematics and Computer Science, Rutgers University, Newark, NJ, USA · Embodied Robotics and Automation Lab, University of South Florida, Tampa, FL, USA
Learning infinite context windows in recurrent architectures via spatial neural computing
Oct 7, 2026Aleix Salvador-Pomarol, Arthur N. Montanari, Earl K. Miller +2Recurrent Neural Networks Long-Context Modeling
UC San Diego · Northwestern University · Massachusetts Institute of Technology
PHBA: Prefix-State Hybrid Block Attention
Oct 6, 2026Ruijie Li, Jiaxi Hu, Shiyu Wang +1Long-Context Modeling Linear Attention
The Hong Kong University of Science and Technology (Guangzhou) · Independent researcher
Memory Depth and Reconstructed Context Width: A Controlled Evaluation of Hierarchical Retrieval
Oct 6, 2026Michael AndreevConversational Memory LLM Evaluation
Independent Researcher
How Far Back Should a Transformer Look? Repetition and Copying in Music Sequence Models
Oct 2, 2026Amir FathiLong-Context Modeling Sequence Modeling
Department of Computing Science University of Alberta Edmonton, AB, Canada
LatentHarness: Learning Latent Actions for Memory and Reasoning via Counterfactual Policy Distillation
Sep 30, 2026Xiaoqiang Wang, Suyuchen Wang, Bang LiuLong-Context Reasoning in Language Models Efficient Language Model Reasoning
Université de Montréal · Mila – Quebec AI Institute
StateTree: Enhancing Long-Term Dialogue Reasoning via Reinforcement Learning
Sep 30, 2026Naen Xu, Wanqing Cui, Yibo Hu +5Temporal Reasoning in Language Models RL for Language Model Reasoning
Zhejiang University · Taobao & Tmall Group of Alibaba
ReMem: Rethinking Perception and Memory in Long-Context Recommendation Agents
Sep 29, 2026Haohao Qu, Yongcheng Jing, Chun Hin Chan +3Long-Context Modeling Recommender Systems
The Hong Kong Polytechnic University Hong Kong · Nanyang Technological University Singapore
SMat-Attention: Structured Long-Context Sequence Modeling
Sep 28, 2026Emile Anand, Abdullah Ateyeh, Archer Wang +1Long-Context Modeling Linear Attention
Massachusetts Institute of Technology
Universality and Generalization of Causal Transformers Across Context Lengths
Sep 28, 2026Takashi Furuya, Maarten V. de Hoop, Gabriel PeyréNeural Network Generalization Transformer Expressivity
Doshisha University, RIKEN AIP · Rice University · CNRS, ENS, PSL Université
Highlight-Then-Summarize: Learning to Compress Evidence for Long-Context Understanding
Sep 25, 2026Zhaoyuan Xia, Qinghongbing Xie, Yung Xiang Hue +7Long-Context Reasoning in Language Models Long-Context Modeling
Peking University · Baidu Inc · Baidu Inc. +1
When Can Agents Forget Their Reasoning? ICLR for Long-Horizon Agent Context Compression
Sep 24, 2026Mingxuan Wang, Fei Luo, Bo Wang +6Long-Context Modeling Long-Horizon LLM Agents
TierFlow Team · Gaoling School of Artificial Intelligence, Renmin University of China · Tsinghua University
Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models
Sep 18, 2026Xize Cheng, Wenxu Jia, Chenyuhao Wen +4Long-Context Language Model Evaluation Long-Context Modeling
Zhejiang University
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
Sep 15, 2026William L. Tong, Aryo Lotfi, Emmanuel Abbe +6Memorization in Language Models Long-Context Modeling
Harvard University · Apple
Residual Vector-based Reconstruction as Long-Context Recall Regardless of Context Window Size
Sep 11, 2026MyungHoon Ryu, XinYu Piao, Jong-Kook KimLLM Memory Long-Context Modeling
Korea University South Korea
ConvMem: Convolutional Memory for Long-Context Reasoning
Sep 9, 2026Hongming Zhang, Zhaozhen Gu, Fengshuo Bai +6Long-Context Reasoning in Language Models Memory-Augmented Language Models
Institute of Automation, Chinese Academy of Sciences
Learning Length-Extrapolatable Recurrent Models
Sep 8, 2026Hanwen JiangBackpropagation Temporal Credit Assignment
Adobe Research
CausalChapter: Improving Long-Video Chaptering with Interventional Dependency Modeling
Sep 8, 2026Xinran Duan, Guozhang Li, Yaoyao Zhong +3Long-Context Modeling Long-Video Understanding
School of Artificial Intelligence, Beijing Normal University

Oct 29, 2026J/K move · Enter open · S save

Topology Survives Shuffling: Robust Long-Context Memory Access Prediction for Graph Analytics

Dongyan Sun, Neelesh Gupta, Rajgopal Kannan, Viktor K. Prasanna

University of Southern California, USA · DEVCOM Army Research Office, USA

Long-Context Modeling Distribution Shift Robustness

Open paper AskPDF arXiv

Graph analytics generates irregular, topology-dependent memory-access streams via indirect traversals, limiting data prefetching. Accurate prediction enables timely prefetches that hide memory latency. However, ML predictors couple short histories with fixed horizons, fitting the training trace's node layout rather than learning transferable graph patterns. We propose LoCoMAP, a long-context representation-learning framework for memory-access prediction with independently controlled observation history and prediction horizon. We evaluate Connected Components (CC), PageRank (PR), and Single-Source Shortest Paths (SSSP) on six synthetic graph families, with Erdos-Renyi (ER) and Grid as random and deterministic controls. We test two interventions: same-family regeneration changes connectivity, while node shuffling disrupts layout but preserves connectivity. Across 18 graph-workload pairs, extending history from 8 to 128 accesses improves F1 by 7.8%, while extending prediction horizon from 8 to 512 improves F1 by 88.3%. For CC, regeneration is less disruptive than shuffling: without retraining, long context retains 95.4% of original F1 and outperforms short context in six families. Under full shuffle, long context improves CC F1 by 33.6% across ER, Barabasi-Albert, Copying Model, and Stochastic Block Model (SBM), with a 2.06x gain on SBM, but not on Watts-Strogatz or Grid, where near-zero delta mass shifts into heavy tails beyond the predictor's range. Long context outperforms short context on six real-world CC graphs. At full shuffle, long-context PR retains 57.6% of original F1 and averages 11.0% above short context. Both SSSP models retain about 30% of original F1 under full shuffle as distance updates access vertex data in a new order. Within these bounds, long-context predictors are robust across related graphs and layouts without retraining, enabling a generalizable graph prefetcher.