LLM Inference Efficiency

Follow this topic Ask about this topic

Recent momentum

-5%

61 papers in the last 28 days · 1.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

21 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

Period ending 2026-09-14

12 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

Period ending 2026-09-07

16 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

541 papers

Latest in LLM Inference Efficiency

Browse all topics →

ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing
May 9, 2026Yongqi An, Chang Lu, Kuan Zhu +5Key-Value Cache Eviction Efficient Long-Context Inference
AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization
May 9, 2026Beshr IslamBouli, David JinLarge Language Model Quantization Weight-Only Quantization
PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding
May 9, 2026Zihao An, Taichi Liu, Ziqiong Liu +3Speculative Decoding LLM Inference Efficiency
Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation
May 8, 2026Joon Ha Kim, Geon-Woo Kim, Anoop Rachakonda +1Profiling LLM Inference Efficiency
RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
May 8, 2026Junkai Zhang, Hang Guo, Luca Benini +1Key-Value Cache Compression Key-Value Cache
Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models
May 8, 2026Boyu Shi, YiCheng Jiang, Chang Liu +3Small Language Models Knowledge Distillation
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
May 8, 2026Ruijie Zhou, Fanxu Meng, Yufei Xu +4Dynamic Sparse Attention Deepseek
SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting
May 8, 2026Weijie Shi, Qiang Xu, Fan Deng +9Speculative Decoding Drafter
Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
May 8, 2026Tho Mai, Joo-Young KimKey-Value Cache Eviction Key-Value Cache
Learning Agent Routing From Early Experience
May 8, 2026Yimin Wang, Jiahao Qiu, Xuan Qi +6Large Language Model Routing LLM Inference Efficiency
Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
May 7, 2026Mohsen Dehghankar, Abolfazl AsudehDynamic Sparse Attention Key-Value Cache
Litespark Inference For CPUs: Ultra-Fast SIMD Framework for Ternary (1.58-bit) Language Models
May 7, 2026Nii Osae Osae Dade, Tony Morri, Moinul Hossain Rahat +2LLM Inference Efficiency Central Processing Units
Towards Distributed Inference of LLMs on a P2P Network
May 7, 2026Shabari S Nair, Krishanu SainiLLM Inference Efficiency Cache
UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification
May 7, 2026Qihang Fan, Huaibo Huang, Zhiying Wu +2Prefill LLM Inference Efficiency
Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
May 7, 2026Muhammad Shahir Abdurrahman, Chun Deng, Azalia Mirhoseini +1Mixture-Of-Expert Inference LLM Inference Efficiency
Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility
May 7, 2026Jungsuk Oh, Hyeseo Jeon, Hyunjune Ji +2Efficient Long-Context Inference LLM Inference Efficiency
Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inference
May 7, 2026Saksham Rathi, Preeti, Mythili VutukuruLLM Inference Efficiency Throughput
HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devices
May 7, 2026Shen Xu, Xiangwen Zhuge, Zhe Xu +3LLM Inference Efficiency Device
HyperLens: Quantifying Cognitive Effort in LLMs with Fine-grained Confidence Trajectory
May 7, 2026Chengda Lu, Xiaoyu Fan, Wei XuLLM Inference Efficiency Confidence Estimation
LLMSpace: Carbon Footprint Modeling for Large Language Model Inference on LEO Satellites
May 7, 2026Lei Jiang, Adrian Ildefonso, Daniel Loveless +1LLM Inference Efficiency Satellite
When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression
May 7, 2026Ruijie Zhang, Haozhe Liang, Da Chang +4Key-Value Cache Eviction Key-Value Cache Compression
ReaComp: Compiling LLM Reasoning into Symbolic Solvers for Efficient Program Synthesis
May 6, 2026Atharva Naik, Yash Mathur, Prakam +2Mock-Interface Synthesis Large Language Model Reasoning
Design Conductor 2.0: An agent builds a TurboQuant inference accelerator in 80 hours
May 6, 2026The Verkor Team, Ravi Krishna, Suresh Krishna +1Accelerator Agent Harness
OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization
May 6, 2026Zhikai Li, Zhen Dong, Xuewen Liu +2Large Language Model Quantization Weight-Only Quantization
A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints
May 6, 2026Chengyi Nie, Nian Si, Zijie ZhouLLM Inference Efficiency Graphics Processing Unit Memory
Parallel Prefix Verification for Speculative Generation
May 5, 2026Yuncheng Yao, Yuxuan Xia, Shengjie Wang +1Speculative Decoding LLM Inference Efficiency
Predict-then-Diffuse: Adaptive Response Length for Compute-Budgeted Inference in Diffusion LLMs
May 5, 2026Michael Rottoli, Subhankar Roy, Stefano ParaboschiLLM Inference Efficiency Diffusion Language Models
QKVShare: Quantized KV-Cache Handoff for Multi-Agent On-Device LLMs
May 5, 2026Pratik Honavar, Tejpratap GVSLKey-Value Cache Quantization LLM Inference Efficiency
Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
May 5, 2026Quanxing Xu, Ling Zhou, Xian Zhong +3Knowledge-Based Visual Question Answering Vision-Language
AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse
May 5, 2026Jie Ou, Jinyu Guo, Shiyao Guo +5Few-Shot Learning In-Context Learning
Two Calls, Two Moments, and the Vote-Accuracy Curve of Repeated LLM Inference
May 5, 2026Yi LiuLLM Inference Efficiency Retrieval Recall
SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection
May 4, 2026Shikhar ShuklaSpeculative Decoding Self-Speculative
WindowQuant: Mixed-Precision KV Cache Quantization based on Window-Level Similarity for VLMs Inference Optimization
May 4, 2026Wei Tao, Xiaoyang Qu, Peiqiang Wang +4Key-Value Cache Quantization Mixed-Precision Quantization
Stochastic Sparse Attention for Memory-Bound Inference
May 3, 2026Kyle Lee, Corentin Delacour, Kevin Callahan-Coray +5Dynamic Sparse Attention LLM Inference Efficiency
Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
May 2, 2026Zijie ZhouLLM Inference Efficiency Heuristics
DARE: Diffusion Language Model Activation Reuse for Efficient Inference
May 1, 2026Natalia Frumkin, Bokun Wang, Hung-Yueh Chiang +3Diffusion Language Models LLM Inference Efficiency
KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
May 1, 2026Shen Han, Yuyang Wu, Junpu Yu +1Key-Value Cache Compression Large Language Model Compression
Position: agentic AI orchestration should be Bayes-consistent
May 1, 2026Theodore Papamarkou, Pierre Alquier, Matthias Bauer +27Decisions LLM Inference Efficiency
Tempus: A Temporally Scalable Resource-Invariant GEMM Streaming Framework for Versal AI Edge
May 1, 2026M. Grailoo, J. Núñez-YáñezHeterogeneous Edge Devices Matrix Multiplication
Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade LLM Inference
May 1, 2026Abdurrahman Javat, Allan KazakovLLM Inference Efficiency Consumer Graphics Processing Units
Context Recycling for Long-Horizon LLM Inference
May 1, 2026Derek ThomasEfficient Long-Context Inference Conversational Benchmarks
Position-Aware Drafting for Inference Acceleration in LLM-Based Generative List-Wise Recommendation
Apr 30, 2026Jiaju Chen, Chongming Gao, Chenxiao Fan +4Speculative Decoding LLM Inference Efficiency
EdgeFM: Efficient Edge Inference for Vision-Language Models
Apr 30, 2026Mengling Deng, Yuanpeng Chen, Sheng Yang +12Heterogeneous Edge Devices LLM Inference Efficiency
Select to Think: Unlocking SLM Potential with Local Sufficiency
Apr 29, 2026Wenxuan Ye, Yangyang Zhang, Xueli An +2Small Language Models Large Language Model Reasoning
LLM-Guided Runtime Parameter Optimization for Energy-Efficient Model Inference
Apr 29, 2026Katelyn Crumpacker, Dimitrios NikolopoulosLLM Inference Efficiency Real-Time Inference
DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference
Apr 29, 2026Bodon Jeong, Hongsu Byun, Youngjae Kim +4Graphics Processing Unit Memory LLM Inference Efficiency
When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?
Apr 29, 2026Tianyu Liu, Yuhao Shen, Xinyi Hu +8Speculative Decoding Kv-Cache Blocks
Efficient, VRAM-Constrained xLM Inference on Clients
Apr 29, 2026Aditya Ukarande, Deep Shekhar, Marc Blackstein +1LLM Inference Efficiency Shard
Breaking the Autoregressive Chain: Hyper-Parallel Decoding for Efficient LLM-Based Attribute Value Extraction
Apr 29, 2026Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani +2Autoregressive Decoding LLM Inference Efficiency
Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
Apr 28, 2026Jiaming Yang, Chenwei Tang, Liangli Zhen +1Key-Value Cache Eviction Eviction
AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devices
Apr 28, 2026Ma Zirui, Fan Zhihua, Li Wenxing +4Speculative Decoding LLM Inference Efficiency
PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference
Apr 27, 2026Ishan Patel, Ishan JoshiKey-Value Cache Compression Key-Value Cache
DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference
Apr 27, 2026Zahra Dehghanighobadi, Asja FischerKey-Value Cache Eviction LLM Inference Efficiency
Less is More: Lightweight Prompt Compression for Question Answering Applications on Edge Devices
Apr 27, 2026Zihuai Xu, Ruofei Hou, Yang Xu +3Large Language Model Compression LLM Inference Efficiency
Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding
Apr 27, 2026Wang Fan, Wei Cao, Xi Zha +5Efficient Long-Context Inference Dynamic Sparse Attention
Hybrid JIT-CUDA Graph Optimization for Low-Latency Large Language Model Inference
Apr 25, 2026Divakar Kumar Yadav, Tian ZhaoLLM Inference Efficiency Inference Latency
Small Language Model Helps Resolve Semantic Ambiguity of LLM Prompt
Apr 25, 2026Zhenzhen Huang, Chaoning Zhang, Fachrina Dewi Puspitasari +4Large Language Model Reasoning Prior Prompt Optimization Techniques
Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
Apr 25, 2026Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park +6Mixture-Of-Expert Inference Mixture-of-Experts Models
Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers
Apr 24, 2026Harri Renney, Fouad Trad, Michael Mattarock +1LLM Inference Efficiency Accelerator
Memanto: Typed Semantic Memory with Information-Theoretic Retrieval for Long-Horizon Agents
Apr 23, 2026Seyed Moein Abtahi, Rasa Rahnema, Hetkumar Patel +3Agent Memory Long-Term Memory

May 9, 2026 · cs.CLJ/K move · Enter open · S save

ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing

Yongqi An, Chang Lu, Kuan Zhu, Tao Yu+4

Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · University of Electronic Science and Technology of China, Chengdu, China · Wuhan AI Research, Wuhan, China · Objecteye Inc., Beijing, China

Large language models (LLMs) face growing challenges in efficient generative inference due to the increasing memory demands of Key-Value (KV) caches, especially for long sequences. Existing eviction methods typically retain KV pairs with high attention weights but overlook the impact of attention redistribution caused by token removal, as well as the spatial-temporal dynamics in KV selection. In this paper, we propose ReST-KV, a robust KV eviction method that combines layer-wise output Reconstruction and Spatial-Temporal smoothing to provide a more comprehensive perspective for the KV cache eviction task. Specifically, ReST-KV formulates KV cache eviction as an optimization problem that minimizes output discrepancies through efficient layer-wise reconstruction. By directly modeling how each token's removal affects the model output, our method naturally captures attention redistribution effects, going beyond simplistic reliance on raw attention weights. To further enhance robustness, we design exponential moving average smoothing to handle temporal variations and an adaptive window-based mechanism to capture spatial patterns. Our method, ReST-KV, significantly advances performance on long-context benchmarks. It surpasses state-of-the-art baselines by 2.58% on LongBench and 15.2% on RULER. Additionally, ReST-KV consistently outperforms existing methods on Needle-in-a-Haystack and InfiniteBench, all while achieving a remarkable 10.61

\times

reduction in decoding latency at 128k context length. The code is publicly available at https://github.com/an-yongqi/rest-kv to facilitate reproducibility and further research.

Key-Value Cache Eviction Efficient Long-Context Inference LLM Inference Efficiency Smoothing

Open paper Ask about it PDF