Efficient Long-Context Inference

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

6 new papers

A weekly snapshot of new work published in Efficient Long-Context Inference.

Inside this field

Focused directions

Focused topic

Efficient Long-Context Inference

Within Efficient Long-Context Inference

229 papersExplore

Focused topic

Context Length

Within Efficient Long-Context Inference

61 papersExplore

Focused topic

Longbench

Within Efficient Long-Context Inference

60 papersExplore

316 papers

Latest in Efficient Long-Context Inference

Browse all topics →

Nectar: Neural Estimation of Cached-Token Attention via Regression
May 10, 2026João Monteiro, Michal Klein, Pierre Ablin +1Softmax Attention Efficient Long-Context Inference
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
May 10, 2026Ngoc Bui, Hieu Trung Nguyen, Arman Cohan +1Key-Value Cache Eviction Efficient Long-Context Inference
Positional LSH: Binary Block Matrix Approximation for Attention with Linear Biases
May 10, 2026Daniel Wolfson, Tal WagnerEfficient Long-Context Inference Locality-Sensitive Hashing
Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Driven
May 10, 2026Jiwei Tang, Zhijing Huang, Xinyu Zhang +5Large Language Model Compression Efficient Long-Context Inference
GRC: Unifying Reasoning-Driven Generation, Retrieval and Compression
May 9, 2026Zhongtao Miao, Qiyu Wu, Yoshimasa TsuruokaMemory-Augmented Large Language Models Longbench
ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference
May 9, 2026Junjie Li, Jiong Lou, Jie LiLLM Inference Efficiency Efficient Long-Context Inference
ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing
May 9, 2026Yongqi An, Chang Lu, Kuan Zhu +5Key-Value Cache Eviction Efficient Long-Context Inference
LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss
May 9, 2026Euntae Choi, Sumin Song, Sungjoo YooLarge Language Model Quantization Extreme Low-Bit Quantization
Scaling Limits of Long-Context Transformers
May 8, 2026Giuseppe Bruno, Shi Chen, Zhengjiang Lin +2Softmax Attention Context Length
RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
May 8, 2026Junkai Zhang, Hang Guo, Luca Benini +1Key-Value Cache Compression Key-Value Cache
An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
May 8, 2026Feiyu Yao, Zhixiong Niu, Xiaqing Li +3Dynamic Sparse Attention Efficient Long-Context Inference
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
May 8, 2026Ruijie Zhou, Fanxu Meng, Yufei Xu +4Dynamic Sparse Attention Deepseek
Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
May 8, 2026Tho Mai, Joo-Young KimKey-Value Cache Eviction Key-Value Cache
Self-Consolidating Language Models: Continual Knowledge Incorporation from Context
May 8, 2026Zekun Wang, Anant Gupta, Zihan Dong +1Efficient Long-Context Inference Consolidation
MiA-Signature: Approximating Global Activation for Long-Context Understanding
May 7, 2026Yuqing Li, Jiangnan Li, Mo Yu +3Signatures Efficient Long-Context Inference
Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility
May 7, 2026Jungsuk Oh, Hyeseo Jeon, Hyunjune Ji +2Efficient Long-Context Inference LLM Inference Efficiency
Training Transformers for KV Cache Compressibility
May 7, 2026Yoav Gelberg, Yam Eitan, Michael Bronstein +2Key-Value Cache Compression Large Language Model Compression
When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression
May 7, 2026Ruijie Zhang, Haozhe Liang, Da Chang +4Key-Value Cache Eviction Key-Value Cache Compression
SCOUT: Active Information Foraging for Long-Text Understanding with Decoupled Epistemic States
May 6, 2026Zhenliang Zhang, Wenqing Wang, Yong Hu +4Context Length Token Communications
MEMAUDIT: An Exact Package-Oracle Evaluation Protocol for Budgeted Long-Term LLM Memory Writing
May 4, 2026Nishant Bhargava, Rodrigo Sobral BarrentoLong-Term Memory Longbench
Retrieval and Multi-Hop Reasoning in 1M-Token Context Windows: Evaluating LLMs on Classical Chinese Text
May 4, 2026Eric H. C. ChowContext Length Local Windows
Evaluating LLMs on Large-Scale Graph Property Estimation via Random Walks
May 2, 2026Sunil Kumar Maurya, Xin LiuBenchmarking Large Language Models Large Language Model Evaluation
On Stable Long-Form Generation: Benchmarking and Mitigating Length Volatility
May 2, 2026Zhitao He, Haolin Yang, Rui Min +2Long-Form Generation Efficient Long-Context Inference
Context Recycling for Long-Horizon LLM Inference
May 1, 2026Derek ThomasEfficient Long-Context Inference Conversational Benchmarks
Structure-Aware Chunking for Tabular Data in Retrieval-Augmented Generation
May 1, 2026Pooja Guttal, Varun Magotra, Vasudeva Mahavishnu +3Chunk Tabular Data
NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus
Apr 30, 2026Enzo S. N. Silva, Pablo B. Costa, Raphael C. Vlasman +8Portuguese Modernbert
AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism
Apr 29, 2026Ahan Gupta, Zhihao Wang, Neel Dani +3Efficient Long-Context Inference Parallel
Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
Apr 28, 2026Jiaming Yang, Chenwei Tang, Liangli Zhen +1Key-Value Cache Eviction Eviction
Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
Apr 27, 2026Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh +7Efficient Long-Context Inference Neural Scaling Laws
Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding
Apr 27, 2026Wang Fan, Wei Cao, Xi Zha +5Efficient Long-Context Inference Dynamic Sparse Attention
Kwai Summary Attention Technical Report
Apr 27, 2026Chenglong Chu, Guorui Zhou, Guowang Zhang +35Efficient Long-Context Inference Quadratic Attention
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Apr 26, 2026DeepSeek-AI, Anyi Xu, Bangcai Lin +316Efficient Long-Context Inference Deepseek
From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priors
Apr 25, 2026Yitian Zhou, Chaoning Zhang, Jiaquan Zhang +6Large Language Model Compression Efficient Long-Context Inference
BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answering
Apr 24, 2026Jinghong Chen, Jingbiao Mei, Guangyu Yang +1Retrieval-Augmented Generation Systems Knowledge-Based Visual Question Answering
SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference
Apr 24, 2026Yuqi Pan, Jinghao Zhuang, Yupeng Feng +16Dynamic Sparse Attention Efficient Long-Context Inference
EngramaBench: Evaluating Long-Term Conversational Memory with Structured Graph Retrieval
Apr 23, 2026Julian AcunaLong-Term Conversational Memory Longbench
Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences
Apr 22, 2026Neehal Tumma, Noel Loo, Daniela RusRecurrent Model Sequence Modeling
ATIR: Towards Audio-Text Interleaved Contextual Retrieval
Apr 22, 2026Tong Zhao, Chenghao Zhang, Yutao Zhu +1Audio Representation Multimodal Retrieval
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
Apr 22, 2026Enshuai Zhou, Yifan Hao, Chao Wang +7Key-Value Cache Compression Large Language Model Compression
Simplified Sparse Attention via Gist Tokens
Apr 22, 2026Yuzhen Mao, Michael Y. Li, Emily B. FoxDynamic Sparse Attention Efficient Long-Context Inference
Absorber LLM: Harnessing Causal Synchronization for Test-Time Training
Apr 22, 2026Zhixin Zhang, Shabo Zhang, Chengcan Wu +2Efficient Long-Context Inference Large Language
'The Order in the Horse's Heart': A Case Study in LLM-Assisted Stylometry for the Discovery of Biblical Allusion in Modern Literary Fiction
Apr 21, 2026Ewan CameronLiterature Stylometric
FG $^2$ -GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control
Apr 21, 2026Pingwei Sun, Yuxuan Hu, Jianchao Tan +6Global Attention Mechanism Efficient Long-Context Inference
Sessa: Selective State Space Attention
Apr 20, 2026Liubomyr HorbatkoEfficient Long-Context Inference Sequence Modeling
Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling
Apr 20, 2026Yujie Chen, Tailai Chen, Yifeng Gao +4Prefill Efficient Long-Context Inference
OPSDL: On-Policy Self-Distillation for Long-Context Language Models
Apr 19, 2026Xinsen Zhang, Zhenkai Ding, Tianjun Pan +4Efficient Long-Context Inference Unsupervised On-Policy Self-Distillation
Long-CODE: Isolating Pure Long-Context as an Orthogonal Dimension in Video Evaluation
Apr 19, 2026Zhijiang Tang, Jiaxin Qi, Bing Zhao +1Longvideobench Large-Scale Video Generation Models
GenericAgent: A Token-Efficient Self-Evolving LLM Agent via Contextual Information Density Maximization (V1.0)
Apr 18, 2026Jiaqing Liang, Jinyi Han, Weijia Li +15Large Language Model Agents Self-Evolving Agents
SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
Apr 18, 2026Junnan Liu, Xinyan Liu, Peifeng Gao +4Efficient Long-Context Inference Large Language Model Decoder
LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks
Apr 18, 2026Xueyao Chen, Jingkai Jia, Tong Yang +3Robotic Manipulation Policies Long-Horizon Planning
AdaSplash-2: Faster Differentiable Sparse Attention
Apr 16, 2026Nuno Gonçalves, Hugo Pitorro, Vlad Niculae +4Dynamic Sparse Attention Efficient Long-Context Inference
LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning
Apr 16, 2026Bowen Ping, Zijun Chen, Tingfeng Hui +4Efficient Long-Context Inference Context Length
KV Cache Offloading for Context-Intensive Tasks
Apr 9, 2026Andrey Bocharnikov, Ivan Ermakov, Denis Kuznedelev +2Kv-Cache Blocks Efficient Long-Context Inference
Internalized Reasoning for Long-Context Visual Document Understanding
Mar 31, 2026Austin VeselkaVisual Document Retrieval Longbench
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
Mar 30, 2026Yufei Xu, Fanxu Meng, Fan Jiang +11Dynamic Sparse Attention Efficient Long-Context Inference
StateLinFormer: Stateful Training Enhancing Long-term Memory in Navigation
Mar 24, 2026Zhiyuan Chen, Yuxuan Zhong, Fan Wang +4Persistent State Long-Term Memory
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Mar 18, 2026Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4Efficient Long-Context Inference Quadratic Attention
LMEB: Long-horizon Memory Embedding Benchmark
Mar 13, 2026Xinping Zhao, Xinshuo Hu, Jiaxin Xu +9Large Language Model Memory Longbench
Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding
Mar 11, 2026Lin Chen, Bolin Ni, Qi Yang +5Cross-Modal Attention Rotary Position Embedding
InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context
Mar 5, 2026Xin Teng, Canyu Zhang, Shaoyi Zheng +3Key-Value Context Length

May 10, 2026 · cs.LGJ/K move · Enter open · S save

Nectar: Neural Estimation of Cached-Token Attention via Regression

João Monteiro, Michal Klein, Pierre Ablin, Marco Cuturi

Apple

Evaluating softmax attention over a fixed long context requires reading every cached key-value pair for each new query token. For a given context (a book, a manual, a legal corpus) the attention output is a deterministic function of the query. We propose Nectar, which fits a compact neural network to this function for queries drawn from a task-relevant distribution. Nectar fits two networks per layer and KV-head: a target network that predicts the attention output and a score network that predicts the log-normalizer. The pair plugs into the standard masked self-attention at inference time, replacing the

O(n)

attention over the cache with a forward pass whose cost does not depend on

n

. Each module carries on the order of

|θ|

parameters per layer and KV-head, typically much smaller than the

2nd

KV-cache footprint at the same granularity. We report experiments on models from 1.7B to 8B parameters across five long-context datasets. The approximation error tracks the next-token accuracy gap to full attention, and allocating capacity non-uniformly across layers reduces that gap in our ablation. Beyond this analysis of metrics, we check that the text generations (following a question prompt) of a model equipped with a Nectar module match in semantic content those obtained by giving the same model access to the full cache.

Softmax Attention Efficient Long-Context Inference Task-Relevant Evidence

Open paper Ask about it PDF