Long-Context Language Model Inference

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 275% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 166

All topics

On-Demand Attention: Language Models Know When to Recall
Sep 17, 2026Haibo Feng, Ruiqi Liang, Hanyang Peng +1Language Model Decoding Long-Context Language Model Inference
Southern University of Science and Technology · Peking University · Peng Cheng Laboratory
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
Sep 17, 2026DeepSeek-AI, :, Anyi Xu +590KV Caching Long-Context Language Model Inference
Long-Lived Characters, Local Inference: Incremental Memory Maintenance for Game NPCs
Sep 16, 2026Zimu XuLong-Context Language Model Inference Persistent Memory for Language Models
University of Bern
ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
Sep 16, 2026Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao +2Long-Context Language Model Inference LLM Inference Scheduling
University of Southern California, Los Angeles, CA, USA
Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference
Sep 13, 2026Tian JinLong-Context Language Model Inference Efficient Language Model Reasoning
Department of Electrical Engineering and Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY
Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?
Sep 9, 2026Fumihiko Tachibana, Daisuke Miyashita, Jun DeguchiRetrieval-Augmented Generation KV Caching
Kioxia Corporation
EvolveScaler: Synthesizing Information-Evolution Contexts via Executable State Machines and Natural-Language Rendering
Sep 8, 2026Ziliang Zhao, Zenan Xu, Shuting Wang +6Long-Context Language Model Inference Temporal Reasoning in Language Models
Gaoling School of Artificial Intelligence, Renmin University of China · LLM Department, Hunyuan Team, Tencent · The Chinese University of Hong Kong
SGD-KV: Summarization Guided KV Cache Compression
Sep 3, 2026Zeyu Liu, Woomin Song, Xuandi Fu +5KV Caching Long-Context Language Model Inference
USC, USA · KAIST, Korea · Amazon AGI, USA
HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models
Sep 2, 2026Renjie Xie, Juncheng Yang, Aoting Hu +4KV Caching Long-Context Language Model Inference
Nanjing University of Posts and Telecommunications · Tylogi AI Lab / TAIL · Anhui University of Technology +2
CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing
Sep 1, 2026Huu Huy Nguyen, Chien Van Nguyen, Franck Dernoncourt +4Long-Context Language Model Inference LLM Inference Acceleration
University of Oregon · Adobe Research · Hanoi University of Science and Technology
Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding
Aug 31, 2026Zhigeng Liu, Zhiyuan Ning, Ruixiao Li +5Long-Context Language Model Inference LLM Inference Acceleration
Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Harbin Institute of Technology, Harbin, China
TuringLLM: Efficiently Scaling Foundation Models Toward Physical AI
Aug 31, 2026Yuheng Zhang, Yizhao Wang, Da Zhu +19Mixture-of-Experts Language Models Long-Context Language Model Inference
Foundation Model Team, Xpeng Inc.
CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
Aug 31, 2026Haoyun Jiang, Haolin Li, Jianwei Zhang +7Long-Context Language Model Inference LLM Inference Acceleration
1CMIC, Shanghai Jiao Tong University · 2Alibaba Group · 3Fudan University
Learning how to Forget: Fine-tuning for Long-Context Sparse Attention
Aug 20, 2026Matthias Seeger, Zeyu Zhang, Vihang Patil +2Fine-Tuning KV Caching
Amazon Web Services · University of Amsterdam · Amazon +1
Self-Indexing Attention for Compression-Compatible Sparse Long-Context LLM Inference
Aug 16, 2026Xu Yang, Jiapeng Zhang, Yuxin Chen +4Long-Context Language Model Inference LLM Inference Acceleration
Hunan University · Tencent
DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference
Aug 9, 2026Asaad AlthoubiKV Caching Long-Context Language Model Inference
Oklahoma State University Stillwater, OK USA
QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
Aug 5, 2026Ayushman Garg, Akshita Gupta, Shaswata Bhattacharya +3KV Caching Long-Context Language Model Inference
Mehta Family School of Data Science and Artificial Intelligence, Indian Institute of Technology Roorkee · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi
Chained Recursive Language Models for Multi-Iteration Reasoning
Aug 5, 2026Purbesh Mitra, Sennur UlukusLong-Context Language Model Inference Language Modeling
University of Maryland
TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning
Aug 4, 2026Wonpyo Park, Seung-won HwangLLM Pruning KV Caching
Seoul National University · Google
ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference
Aug 3, 2026Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino +1Long-Context Language Model Inference LLM Inference Acceleration
Graduate School of Informatics, Nagoya University · Information Technology Center, Nagoya University
AnchorKV: Anchor-Residual KV Cache Compression
Aug 3, 2026Malik Khalaf, Yara Shamshoum, Nitzan Hodos +2KV Caching Long-Context Language Model Inference
Department of Computer Science, Technion - Israel Institute of Technology
Opt.Gear Technical Report
Aug 2, 2026Juneyoung Park, Youngwook KwonOn-Device Language Model Inference Long-Context Language Model Inference
Opt.Gear Team
S $^4$ R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching
Aug 1, 2026Jialong Han, You Wu, Kewei TuKV Caching Long-Context Language Model Inference
School of Information Science and Technology, ShanghaiTech University · Shanghai Engineering Research Center of Intelligent Vision and Imaging
CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residuals
Jul 30, 2026Hanzuo Liu, Xuan Qi, Chunyu Liu +6Memory-Augmented Language Models Long-Context Language Model Inference
Tsinghua University · Tencent
A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
Jul 30, 2026Yuesong Liu, Yuan Zeng, Min Lyu +3Long-Context Language Model Inference Speculative Decoding
University of Science and Technology of China
Recall Before You Rank: Similarity-Guided Top- $K$ Reuse for Efficient Long-Context Attention
Jul 30, 2026Wenshuai Yao, Wenyong Zhou, Hanyong Shao +5Long-Context Language Model Inference Efficient Attention
School of Integrated Circuits, Peking University, Beijing, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China · School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China
CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention
Jul 28, 2026Yufei Xue, Lin Niu, Hong Liu +6Long-Context Language Model Inference LLM Inference Acceleration
1Tencent · 2The Hong Kong University of Science and Technology
LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding
Jul 27, 2026Junsung HwangLow-Rank Attention KV Caching
RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention
Jul 24, 2026Anderson R. SantosLong-Context Language Model Inference Memory-Efficient Inference
Faculty of Computing, Federal University of Uberlândia (UFU), Brazil.
Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
Jul 23, 2026Alagappan ValliappanMulti-Token Prediction Long-Context Language Model Inference
NVIDIA

Sep 17, 2026, cs.CLJ/K move · Enter open · S save

On-Demand Attention: Language Models Know When to Recall

Haibo Feng, Ruiqi Liang, Hanyang Peng, Shiqi Yu

Southern University of Science and Technology · Peking University · Peng Cheng Laboratory

Language Model Decoding Long-Context Language Model Inference LLM Inference Acceleration Attention Mechanisms

Open paper AskPDF arXiv

Reasoning and agentic workloads increasingly demand efficient long-context inference. Yet full-attention decoding reads the growing history at every step, regardless of its benefit to the next prediction. We show that a pretrained model's decoding states already contain information predictive of this benefit, before the global read. Building on this finding, we introduce On-Demand Attention (ODA), a local-first decoding method that uses a lightweight recall head to selectively invoke global attention as its predicted benefit changes during generation. ODA trains only the recall head, leaving pretrained weights unchanged and the complete historical KV cache available for future recall. We further implement GPU-side conditional execution in vLLM, translating reduced global reads into practical decoding speedups over full attention at long context lengths. Experiments across Qwen and Gemma models, including hybrid-attention backbones, show that selective recall recovers most of the performance lost under local attention while substantially reducing global reads. These findings support long-context inference in which pretrained models guide their own access to the information they retain.

Momentum

Related topics

Latest papers 166