LLM Inference Acceleration

LLM: Large Language Model

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

Fast KV Compaction via Attention Matching
Feb 18, 2026Adam Zweiger, Xinghong Fu, Han Guo +1KV-Cache Management LLM Context Compression
Massachusetts Institute of Technology.
CLAA: Cross-Layer Attention Aggregation for Accelerating LLM Prefill
Feb 17, 2026Bradley McDanel, Steven Li, Harshit KhaitanCross-Layer Attention LLM Inference Acceleration
Franklin and Marshall College · Meta Reality Labs
MoE-Spec: Expert Budgeting for Efficient Speculative Decoding
Feb 17, 2026Bradley McDanel, Steven Li, Sruthikesh Surineni +1Speculative Decoding LLM Inference Acceleration
Franklin and Marshall College · Meta Reality Labs
Mobility-Aware Cache Framework for Scalable LLM-Based Human Mobility Simulation
Feb 17, 2026Hua Yan, Heng Tan, Yingxue Zhang +1Human Behavior Simulation LLM Inference Acceleration
Lehigh University Bethlehem, USA · State University of New York at Binghamton Binghamton, USA
WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsity
Feb 16, 2026Lei Chen, Yuan Meng, Xiaoyu Zhan +2Activation Sparsity LLM Inference Acceleration
Tsinghua University
QTALE: Quantization-Robust Token-Adaptive Layer Execution for LLMs
Feb 11, 2026Kanghyun Noh, Jinheon Choi, Yulhwa KimLLM Quantization LLM Compression
Department of Semiconductor Convergence Engineering, Sungkyunkwan University · Department of Electrical and Computer Engineering, Sungkyunkwan University · Department of Semiconductor Systems Engineering, Sungkyunkwan University
Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference
Feb 9, 2026Yifei Gao, Lei Wang, Rong-Cheng Tu +3Self-Attention KV Caching
University College London, London, UK · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences (CAS), Shenzhen 518055, China · The Chinese University of Hong Kong +1
ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
Feb 8, 2026Jitai Hao, Qiang Huang, Yaowei Wang +2Self-Attention KV Caching
Harbin Institute of Technology (Shenzhen), China
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
Feb 5, 2026Jingze Shi, Zhangyang Peng, Yizhang Zhu +3Mixture-of-Experts Inference LLM Inference Acceleration
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence
TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference
Feb 5, 2026Jiyoung Park, Hankyu Jang, Changseok Song +1Adaptive Inference LLM Inference Serving
Moreh, Inc.
Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States
Feb 3, 2026Ximing Dong, Shaowei Wang, Dayi Lin +2Speculative Decoding LLM Inference Acceleration
Centre for Software Excellence, Huawei, Canada · Department of Computer Science, University of Manitoba, Canada · School of Computing, Queen’s University, Canada
MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models
Feb 3, 2026Maximilian Kleinegger, Elvir Crnčević, Dan AlistarhLLM Quantization LLM Inference Acceleration
1Vienna University of Technology, Vienna, Austria · Institute of Science & Technology Austria (ISTA), Vienna, Austria · 3Red Hat AI, Boston, USA
STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs
Feb 2, 2026Weikang Meng, Liangyu Huo, Yadan Luo +4LLM Inference Acceleration Linear Attention
SMULL Group, Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · UQMM Lab, University of Queensland +1
ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure
Feb 1, 2026Jie Deng, Shining Liang, Jun Li +2Efficient Language Model Reasoning LLM Inference Acceleration
University of Chinese Academy of Sciences, Beijing, China · 2Microsoft, Beijing, China · 3Tongji University, Shanghai, China
Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoning
Jan 30, 2026Arvind Mahankali, Kaiyue Wen, Tengyu MaMulti-Agent LLM Systems CoT Reasoning
Stanford University
Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
Jan 25, 2026Jang-Hyun Kim, Dongyoon Han, Sangdoo YunKV-Cache Eviction KV-Cache Compression
NAVER AI Lab.
Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference
Jan 19, 2026Zimeng Wu, Donghao Wang, Chaozhe Jin +2LLM Inference Acceleration Long-Context LLM Inference
School of Computer Science and Engineering, Beihang University, Beijing, China · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
Jan 12, 2026Haoqian Meng, Yilun Luo, Yafei Zhao +3LLM Quantization 4-Bit Quantization
School of Computer Science and Technology, Tianjin University
JuDi: Revisiting Judge Decoding from First Principles via Training-Free Distributional Divergence
Jan 8, 2026Shengyin Sun, Yiming Li, Renxi Liu +5Speculative Decoding LLM Inference Acceleration
City University of Hong Kong · Huawei Technologies
Sandwich Reasoning: An Answer-Reasoning-Answer Approach for Low-Latency Query Correction
Jan 7, 2026Chen Zhang, Kepu Zhang, Jiatong Zhang +2Efficient Language Model Reasoning Query Reformulation
Gaoling School of Artificial Intelligence, Renmin University of China
Entropy-Aware Token Rejection for Improving Speculative Decoding
Dec 29, 2025Tiancheng Su, Meicong Zhang, Guoxiu HeSpeculative Decoding LLM Inference Acceleration
School of Economics and Management East China Normal University
Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
Dec 18, 2025Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra +1Self-Attention Long-Context Language Model Inference
Microsoft Research India.
xGR: Efficient Generative Recommendation Serving at Scale
Dec 12, 2025Qingxiao Sun, Tongxuan Liu, Shen Zhang +13LLM Serving LLM Inference Acceleration
Beihang University · JD Company · University of Science and Technology Beijing +1
Cerium: A Multi-GPU Framework for Terabyte-Scale Encrypted Inference
Dec 12, 2025Siddharth Jayashankar, Joshua Kim, Michael B. Sullivan +2Homomorphic Encryption GPU Kernel Optimization
Carnegie Mellon University · UT Austin · NVIDIA
Block Sparse Flash Attention
Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3GPU Acceleration Self-Attention
Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel
Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement
Dec 5, 2025Nils Strassenburg, Boris Glavic, Tilmann RablCost-Aware Inference Small Language Models
Hasso Plattner Institute, Uni Potsdam · University of Illinois Chicago
Data-Free Pruning of Self-Attention Layers in LLMs
Dec 3, 2025Dhananjay Saikumar, Blesson VargheseLLM Inference Acceleration LLM Compression
School of Computer Science University of St Andrews United Kingdom
ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models
Nov 24, 2025Long Lian, Sida Wang, Felix Juefei-Xu +7Parallel Reasoning in Language Models RL for Language Model Reasoning
Meta Superintelligence Labs (MSL), Menlo Park, CA, United States · UC Berkeley, Berkeley, CA, United States · UCSF, San Francisco, CA, United States
DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architectures
Nov 19, 2025Peiming Yang, Sankeerth Durvasula, Ivan Fernandez +4Compute-in-Memory LLM Inference Acceleration
University of Toronto & Vector Institute · Barcelona Supercomputing Center · ETH Zürich +2
Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference
Nov 19, 2025Kexin Chu, Dawei Xiang, Zixu Shen +3Mixed-Precision Quantization GPU Acceleration
University of Connecticut · University of California, Santa Cruz · Independent Researcher

Feb 18, 2026J/K move · Enter open · S save

Fast KV Compaction via Attention Matching

Adam Zweiger, Xinghong Fu, Han Guo, Yoon Kim

Massachusetts Institute of Technology.

KV-Cache Management LLM Context Compression KV-Cache Compression Long-Context Language Modeling LLM Inference Acceleration

Open paper AskPDF arXiv

Scaling language models to long contexts is often bottlenecked by the size of the key-value (KV) cache. In deployed settings, long contexts are typically managed through compaction in token space via summarization. However, summarization can be highly lossy, substantially harming downstream performance. Recent work on Cartridges has shown that it is possible to train highly compact KV caches in latent space that closely match full-context performance, but at the cost of slow and expensive end-to-end optimization. This work describes an approach for fast context compaction in latent space through Attention Matching, which constructs compact keys and values to reproduce attention outputs and preserve attention mass at a per-KV-head level. We show that this formulation naturally decomposes into simple subproblems, some of which admit efficient closed-form solutions. Within this framework, we develop a family of methods that significantly push the Pareto frontier of compaction time versus quality, achieving up to 50x compaction in seconds on some datasets with little quality loss.