LLM Inference Acceleration

LLM: Large Language Model

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning
Mar 21, 2026Dong Liu, Yanxuan Yu, Ben Lengerich +1Attention Mechanisms Memory-Efficient Attention
University of California, Los Angeles, Los Angeles, California, USA · Columbia University, New York, New York, USA · University of Wisconsin-Madison, Madison, Wisconsin, USA
BEAVER: A Training-Free Hierarchical Prompt Compression Method via Structure-Aware Page Selection
Mar 20, 2026Zhengpei Hu, Kai Li, Dapeng Fu +4LLM Context Compression Prompt Compression
School of Computer Technology and Application, Qinghai University · Tsinghua University · Ant Group Security and Intelligence Laboratory (SIL)
SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
Mar 19, 2026Shenggui Li, Chao Wang, Yikai Zhu +14Speculative Decoding LLM Inference Acceleration
Nanyang Technological University · SpecForge Team · Meituan +2
Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing
Mar 18, 2026Raghavv Goel, Mukul Gagrani, Mingu Lee +1Multi-Token Prediction LLM Inference Acceleration
Qualcomm AI Research
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Mar 18, 2026Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4Self-Attention Long-Context Language Modeling
Department of Electrical and Computer Engineering, Purdue University, USA · AWS Agentic AI, USA
ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression
Mar 18, 2026Ruibo Fan, Xiangrui Yu, Xinglin Pan +5Model Compression LLM Compression
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Harbin Institute of Technology, Shenzhen, Shenzhen, China · The Hong Kong University of Science and Technology, Hong Kong, Hong Kong SAR
VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
Mar 17, 2026Yixuan Wang, Qingyu Shi, Jiayu Zhou +3KV-Cache Compression KV-Cache Quantization
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory +2
Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective
Mar 17, 2026Noppanat Wadlom, Junyi Shen, Yao LuLLM Serving LLM Agent Workflow Optimization
National University of Singapore Singapore
MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment
Mar 16, 2026Hanxian Huang, Igor Fedorov, Andrey Gromov +14On-Device Language Model Inference LLM Inference Acceleration
Meta AI
FlashSampling: Fast and Memory-Efficient Exact Sampling
Mar 16, 2026Tomas Ruiz, Zhen Qin, Yifan Zhang +3LLM Inference Acceleration GPU Kernel Optimization
LMU Munich · FlashSampling · Princeton University
FlashHead: Efficient Drop-In Replacement for the Classification Head in Language Model Inference
Mar 15, 2026Wilhelm Tranheden, Shahnawaz Ahmed, Devdatt Dubhashi +2LLM Inference Acceleration Efficient Language Model Inference
Embedl Gamla Almedalsvägen 21, 412 63 Gothenburg, Sweden
SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Merging
Mar 15, 2026Shunlong Wu, Hai Lin, Shaoshen Chen +5KV-Cache Compression Memory-Efficient Inference
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · Seoul National University, South Korea
Self-Indexing KVCache: Predicting Sparse Attention from Compressed Keys
Mar 15, 2026Xu Yang, Jiapeng Zhang, Dongyang Zhao +2Binary Quantization LLM Inference Acceleration
College of Computer Science and Electronic Engineering, Hunan University, Changsha, China · The Ministry of Education Key Laboratory of “Fusion Computing of Supercomputing and Artificial Intelligence” China · Shenzhen Research Institute, Hunan University, Shenzhen, China
SVD Contextual Sparsity Predictors for Fast LLM Inference
Mar 14, 2026Georgii Serbin, Kirill Koshkin, Zhongao Sun +2LLM Inference Acceleration
Huawei Technologies Ltd. · Moscow State University, Russia.
98 $\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router
Mar 13, 2026Xunzhuo Liu, Bowei He, Xue Liu +3LLM Routing LLM Serving
vLLM Semantic Router Project · MBZUAI · McGill University +2
When Drafts Evolve: Speculative Decoding Meets Online Learning
Mar 13, 2026Yu-Yang Qian, Hao-Cong Wu, Yichao Fu +2Speculative Decoding LLM Inference Acceleration
State Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · University of California, San Diego
Dynamic Sparse Attention: Access Patterns and Architecture
Mar 13, 2026Noam LevyLLM Inference Acceleration Sparse Attention
Intel Corporation.
TERMINATOR: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoning
Mar 13, 2026Alliot Nagle, Jakhongir Saydaliev, Dhia Garbaya +3Early Stopping for Language Model Reasoning Efficient Language Model Inference
UT Austin · EPFL · ENS Paris-Saclay +1
Accelerating Suffix Jailbreak attacks with Prefix-Shared KV-cache
Mar 12, 2026Xinhai Wang, Shaopeng Fu, Shu Yang +3Jailbreak Attacks LLM Jailbreak Attacks
King Abdullah University of Science and Technology (KAUST) · Zhejiang University
IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
Mar 12, 2026Yushi Bai, Qian Dong, Ting Jiang +5LLM Inference Acceleration Sparse Attention
Tsinghua University
Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stability
Mar 12, 2026Xingyu Xie, Zhaochen Yu, Yue Liao +3LLM Inference Acceleration Long-Context Language Model Inference
National University of Singapore · ByteDance
Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Language
Mar 12, 2026Remigiusz Kinas, Paweł Kiszczak, Sergio P. Perez +4LLM Compression Language Modeling
Bielik.AI, Ingenix.ai · Bielik.AI, Vstorm · NVIDIA +3
AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization
Mar 12, 2026Qiyang Li, Rui Kong, Yuchen Li +5LLM Inference Acceleration Low-Rank Adaptation
Baidu Inc. · Shanghai Jiao Tong University
LongFlow: Efficient KV Cache Compression for Reasoning Models
Mar 12, 2026Yi Su, Zhenxu Tian, Dan Qiao +3KV-Cache Compression LLM Inference Acceleration
School of Computer Science and Technology, Soochow University, China · ByteDance
LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation
Mar 11, 2026Jinwoo Ahn, Ingyu Seong, Akhil Kedia +4KV-Cache Eviction Long-Context Language Model Inference
Samsung Research
TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly
Mar 11, 2026Toshiaki Koike-Akino, Jing Liu, Ye WangLLM Quantization Neural Network Quantization
Mitsubishi Electric Research Laboratories (MERL), 201 Broadway, Cambridge, MA 02139, USA
Compiler-First State Space Duality and Portable $O(1)$ Autoregressive Caching for Inference
Mar 10, 2026Cosmo Santoni, Anmol ThaparEfficient Inference LLM Inference Acceleration
Imperial College London
Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation
Mar 10, 2026Luxi Lin, Zhihang Lin, Zhanpeng Zeng +5Speculative Decoding LLM Inference Acceleration
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Innovation Institute +1
ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios
Mar 10, 2026Xinyi Hu, Yuhao Shen, Baolin Zhang +6Speculative Decoding LLM Inference Acceleration
Qwen Applications Business Group of Alibaba · Zhejiang University
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
Mar 10, 2026Yinpeng Wu, Yitong Chen, Lixiang Wang +3LLM Serving On-Device Language Model Inference
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University

Mar 21, 2026J/K move · Enter open · S save

MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning

Dong Liu, Yanxuan Yu, Ben Lengerich, Ying Nian Wu

University of California, Los Angeles, Los Angeles, California, USA · Columbia University, New York, New York, USA · University of Wisconsin-Madison, Madison, Wisconsin, USA

Attention Mechanisms Memory-Efficient Attention Long-Context Language Modeling KV-Cache Management LLM Inference Acceleration Memory-Augmented Language Models

Open paper AskPDF arXiv

As long-context language modeling becomes increasingly important, the cost of maintaining and attending to large Key/Value (KV) caches grows rapidly, becoming a major bottleneck in both training and inference. While prior works such as Multi-Query Attention (MQA) and Multi-Latent Attention (MLA) reduce memory by sharing or compressing KV features, they often trade off representation quality or incur runtime overhead. We propose Memory-Keyed Attention (MKA), a hierarchical attention mechanism that integrates multi-level KV caches (local, session, and long-term) and learns to route attention across them dynamically. We further introduce Route-Fused MKA (FastMKA), a broadcast-routed variant that fuses memory sources before attention computation for improved efficiency. Experiments on different sequence lengths show that FastMKA achieves a favorable accuracy-efficiency trade-off: comparable perplexity to MLA while achieving up to 5x faster training throughput and 1.8x lower evaluation latency. These results highlight MKA as a practical and extensible framework for efficient long-context attention.