Long-Context Inference

Follow this topic Ask about this topic

Long-context inference asks how language models can use information spread across lengthy documents, conversations, or corpora, not merely accept a large number of tokens. Approaches include sparse attention and KV-cache compression, retrieval and compact memory, chunked or multi-agent processing, and test-time adaptation. Benchmarks test needle retrieval, multi-hop question answering, information aggregation, and reasoning over dispersed evidence. A central challenge is that longer context windows do not guarantee effective reasoning, while reducing memory and computation can omit evidence needed for correct answers.

Latest papers

52 papers8 papers in the last four weeks, up 167% on the four weeks before. 0.0% of all new papers.

Local Inference of LLMs on Long Contexts: A Data Systems Perspective
Oct 20, 2026Mattias Dürrmeier, Alberto LernerLLM Inference Long-Context Inference
University of Fribourg, Fribourg, Switzerland · Computing Flows GmbH, Fribourg, Switzerland
Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning
Oct 15, 2026Shiwen Qin, Cameron Barker, Henry Gouk +4Long-Context Reasoning in Language Models Long-Context Language Model Evaluation
University of Edinburgh, Edinburgh, United Kingdom
MemDuet: Learning Dual-Process Memory for Long-Context Reasoning
Oct 14, 2026Jiangcheng Song, Haodong Jing, Yulin Mu +7Dual-Process Reasoning Memory-Augmented Language Model Agents
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Faculty of Electronic and Information Engineering, Xi’an Jiaotong University · Imperial College London
CacheLink: Functional KV Transport for Zero-Prefill Cross-Model Context Handoff
Oct 11, 2026Haomiao Qiu, Yining Tang, Zijia Song +3LLM Serving KV-Cache Reuse
Harbin Institute of Technology, Shenzhen · Great Bay University · Zhejiang University
LongHarness: Advancing Agentic Reasoning over 100M-Token Corpora via Stateful Evidence Management
Oct 9, 2026Songtao Huang, Zhijie Zhong, Xiaohan He +8LLM Agent Harnesses Long-Context Inference
Fudan University · Shanghai Artificial Intelligence Laboratory · Shanghai Jiaotong University +3
Read What Matters: Query-Adaptive Quantization for KV Caches
Oct 8, 2026Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian +1KV-Cache Quantization KV-Cache Compression
Amazon · University of California, Berkeley
UNREAL: Unifying Retrieval and Long-Context with a Single Model
Oct 6, 2026Edan Kinderman, Elad Hoffer, Yochai Blau +4Retrieval-Augmented Generation Knowledge Augmentation for Language Models
NVIDIA · Technion
LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning
Sep 29, 2026Quang Hieu Pham, Thuy Duong Nguyen, Jocelyn Qiaochu Chen +1Long-Context Language Model Evaluation Long-Context Language Model Inference
University of Alberta · Alberta Machine Intelligence Institute (Amii)
Realize What Matters: Principled Context Representation for Large-Scale Reasoning
Sep 23, 2026Michael Theologitis, Dean Light, Shuyue Stella Li +3Long-Context Inference LLM Context Management
University of Washington
PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agents
Sep 6, 2026Kun Li, Zexuan Qiu, Tianhua Zhang +2Long-Context Inference Multi-Agent LLMs
The Chinese University of Hong Kong
Trace as State: Reasoning Traces as Conditional States for Long-Context Transformers
Sep 2, 2026Xu Zou, Jie TangLong-Context Inference CoT Reasoning
Tsinghua University
Sliding-window beats linear attention
Aug 28, 2026Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron +1Linear Attention KV Caching
Microsoft Applied Sciences Group (ASG) · Independent
Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning
Aug 19, 2026Zhu Zhang, Jixun Wang, Xiaoang Xu +6On-Policy Distillation Knowledge Distillation
Tsinghua University · OpenBMB · Beijing University of Posts and Telecommunications
Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression
Aug 5, 2026Zhengpei Hu, Kai Li, Dapeng Fu +5LLM Compression Long-Context Inference
School of Computer Technology and Application, Qinghai University · Tsinghua University · Ant Group Security and Intelligence Laboratory (SIL)
SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering
Jul 31, 2026Maryam Haghifam, Jason Cong, Yizhou SunMemory-Augmented Language Models Long-Context QA
University of California, Los Angeles, USA
ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression
Jul 31, 2026Yuhang Zhan, Lisi Chen, Shuo ShangSoftmax Attention KV Caching
University of Electronic Science and Technology of China
C $^2$ KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
Jul 20, 2026Chuheng Du, Junyi Chen, Hanlin Tang +7KV Caching LLM Inference Acceleration
School of Computer Science Shanghai Jiao Tong University Shanghai, China · Alibaba Group Hangzhou, China
RECON: Benchmarking Agent Memory for Compositional Reasoning over Long Contexts
Jul 18, 2026Mihir Shriniwas AryaCompositional Reasoning LLM Agent Memory
Department of Computer Science and Engineering RV College of Engineering
Self-Guided Test-Time Training for Long-Context LLMs
Jul 10, 2026Xinyu Zhu, Zhe Xu, Xiaohan Wei +10Long-Context Language Modeling Test-Time Training
Meta AI · University of Virginia
WILDTRACE: Benchmarking Natural Evidence Trails in Long-Context Reasoning
Jul 10, 2026Zixin Chen, Peng Liu, Haobo Li +7Evidence-Grounded Reasoning Long-Context QA
Hong Kong University of Science and Technology · Qwen Team, Alibaba Group · *Work done during internship at Qwen Team
Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
Jul 9, 2026Siddhartha Jain, Ameya VelingkerLLM Evaluation Long-Context Language Model Evaluation
NVIDIA
The Key to Going Linear: Analysis-Driven Transformer Linearization
Jul 8, 2026Anna Kuzina, Paul N. Whatmough, Babak Ehteshami BejnordiSoftmax Attention LLM Inference Acceleration
Qualcomm AI Research∗ · Qualcomm AI Research
ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning
Jul 2, 2026Yanjun Zhao, Ruizhong Qiu, Tianxin Wei +6Long-Context Inference Long-Context Reasoning in Language Models
University of Illinois Urbana-Champaign
Evidence-State Rewards for Long-Context Reasoning
Jul 2, 2026Ya Gao, Pekka MarttinenProcess Reward Models RL for Language Model Reasoning
1Aalto University
Information-Aware KV Cache Compression for Long Reasoning
Jun 25, 2026Jushi Kai, Zhuiri Xiao, Alexandra Birch +1KV Caching Long-Context Language Model Inference
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Jiao Tong University · School of Informatics, University of Edinburgh
Randomized YaRN Improves Length Generalization for Long-Context Reasoning
Jun 22, 2026Manas Mehta, Fangcong Yin, Greg DurrettLong-Context Language Modeling LLM Fine-Tuning
New York University
Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
Jun 17, 2026Xiaoyue Xu, Sikui Zhang, Xiaorong Wang +2Training Data Curation RL for Language Model Reasoning
OpenBMB · Tsinghua University
Recursive Agent Harnesses
Jun 11, 2026Elias Lumer, Sahil Sen, Kevin Paul +1Tool-Augmented Language Model Agents LLM Agent Orchestration
PricewaterhouseCoopers, U.S.
HMARS: A Hierarchical Multi-Agent Memory System for Long-Context Reasoning
Jun 3, 2026Zeju Li, Ziyang Zheng, Yizhou Zhou +1Agent Memory Multi-Agent LLM Systems
The Chinese University of Hong Kong · Independent Researcher
LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
May 31, 2026Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li +1LLM Compression LLM Fine-Tuning
SambaNova Systems, Inc. San Jose, CA, USA

Oct 20, 2026J/K move · Enter open · S save

Local Inference of LLMs on Long Contexts: A Data Systems Perspective

Mattias Dürrmeier, Alberto Lerner

University of Fribourg, Fribourg, Switzerland · Computing Flows GmbH, Fribourg, Switzerland

LLM Inference Long-Context Inference On-Device Language Model Inference KV-Cache Management

Open paper AskPDF arXiv

Large language models can solve complex problems across a variety of subjects, such as mathematics and coding. For such complex analytical tasks, models with a long context perform better. Serving models with long context is currently mostly done on powerful systems with multiple GPUs, which are too expensive for most users. An alternative is local inference, or serving a model on a user's own machine, which removes per-token costs and rate limits, besides keeping sensitive data private and giving users full control over the models they run. The common understanding, however, is that local LLMs need compression (e.g., quantization) to run on a single GPU. In this paper, we set out to study the limitations that prevent serving long-context models for single users, besides the capacity issue. Rather than characterizing a single model on one GPU, we analyze how different models behave across several GPU generations. This lets us uncover not just individual bottlenecks but broader trends in where inefficiencies arise. Our analysis confirms that, as expected, self-attention dominates the runtime for long-context inference. More unexpectedly, our evaluation reveals that the current structure of self-attention leaves key GPU resources significantly underutilized, and the newest GPU is often the least well served. Many of these findings are fundamentally data management problems, from paging the KV cache to improving cache utilization and synchronization in the GPU, which is why we bring this study to the Data Systems community. To engage this community, we present our findings as a series of experimental observations and formulate recommendations for future work.