LLM Inference Acceleration

LLM: Large Language Model

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

AQPIM: Breaking the PIM Capacity Wall for LLMs with In-Memory Activation Quantization
Apr 20, 2026Kosuke Matsushima, Yasuyuki Okoshi, Masato Motomura +1Compute-in-Memory Long-Context Language Model Inference
Institute of Science Tokyo†
Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling
Apr 20, 2026Yujie Chen, Tailai Chen, Yifeng Gao +4Self-Attention Long-Context Language Model Inference
Shanghai Jiao Tong University · University of California, San Diego · Carnegie Mellon University
Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM
Apr 20, 2026Sravanth Kodavanti, Sowmya Vajrala, Srinivas Miriyala +13On-Device Language Model Inference LLM Inference
Samsung Research Institute Bangalore, India · Samsung Electronics, Suwon, South Korea
M100: An Orchestrated Dataflow Architecture Powering General AI Computing
Apr 20, 2026Yan Xie, Changkui Mao, Changsong Wu +34AI Accelerator Inference LLM Inference Acceleration
Li Auto
DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models
Apr 20, 2026You-Liang Huang, Xinhao Huang, Chengxi Liao +1LLM Inference Parallel LLM Inference
Boston University · Boston, Massachusetts, USA · HKUST (Guangzhou) +2
Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
Apr 19, 2026Sanjeev Rao GanjihalKV-Cache Offloading KV Caching
Independent Researcher
Hive: A Multi-Agent Infrastructure for Algorithm- and Task-Level Scaling
Apr 19, 2026Zizhang Luo, Yuhao Luo, Youwei Xiao +3LLM Inference Efficiency Multi-Agent LLM Systems
Peking University Beijing, China
CRISP: Compressing Redundancy in Chain-of-Thought via Intrinsic Saliency Pruning
Apr 19, 2026Yangsong Lan, Hongliang Dai, Piji LiLLM Pruning LLM Compression
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China.
Probabilistic Programs of Thought
Apr 19, 2026Poorva Garg, Renato Lui Geh, Daniel Israel +3LLM-Based Program Synthesis Efficient Language Model Reasoning
University of California Los Angeles · Allen Institute for AI
UCCL-Zip: Lossless Compression Supercharged GPU Communication
Apr 19, 2026Shuang Ma, Chon Lam Lao, Zhiying Xu +8GPU Kernel Optimization GPU Acceleration
UC Davis · Harvard University · Amazon Web Services +1
Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon
Apr 18, 2026Sai VegasenaAI Accelerator Inference Self-Attention
Ensue
Step-GRPO: Internalizing Dynamic Early Exit for Efficient Reasoning
Apr 18, 2026Benteng Chen, Weida Wang, Shufei Zhang +2RL for Language Model Reasoning LLM Inference Acceleration
East China Normal University · Shanghai AI Laboratory · Fudan University +1
SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
Apr 18, 2026Junnan Liu, Xinyan Liu, Peifeng Gao +4GPU Acceleration Self-Attention
Harbin Institute of Technology (Weihai) · University of Hong Kong, Hong Kong, China · City University of Hong Kong, Hong Kong, China
Cut Your Losses! Learning to Prune Paths Early for Efficient Parallel Reasoning
Apr 17, 2026Jiaxi Bi, Tongxu Luo, Wenyu Du +2LLM Inference Acceleration Large Reasoning Models
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · DualityRL
Sparse Prefix Caching for Hybrid and Recurrent LLM Serving
Apr 17, 2026Mikhail Shirokikh, Sergey NikolenkoLLM Serving KV-Cache Management
EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptation
Apr 17, 2026Shuyu Zhang, Lingfeng Pan, Qicheng Wang +6Speculative Decoding LLM Inference Acceleration
School of Computer Science and Technology Xidian University Xi’an, China
Faster LLM Inference via Sequential Monte Carlo
Apr 17, 2026Yahya Emara, Mauricio Barba da Costa, Chi-Chih Chang +4Speculative Decoding LLM Inference Acceleration
Cornell University · Makora · MIT +1
Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU
Apr 16, 2026Jevin Jiang, Ying Chen, Blake A. Hechtman +2Efficient Transformer Inference AI Accelerator Inference
Google
From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning
Apr 16, 2026Kiran Purohit, Ramasuri Narayanam, Soumyabrata PalLanguage Model Self-Verification LLM Answer Verification
IIT Kharagpur · Adobe Research
ARGUS: Agentic GPU Optimization Guided by Data-Flow Invariants
Apr 16, 2026Haohui Mai, Xiaoyan Guo, Xiangyun Ding +7GPU Kernel Optimization GPU Acceleration
UC Riverside5 · UCAS4 · Tsinghua University2 +2
MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration
Apr 16, 2026Xinyu Liu, Xin Liu, Bo Jin +8Multi-Token Prediction KV Caching
School of Computer Science and Engineering, Northeastern University, China · Meituan Inc. · NiuTrans Research, Shenyang, China
RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding
Apr 16, 2026Zihong Zhang, Zuchao Li, Lefei Zhang +2Speculative Decoding LLM Inference Acceleration
School of Artiﬁcial Intelligence, Wuhan University, Wuhan, China · School of Computer Science, Wuhan University, Wuhan, China · School of Information Management, Wuhan University, Wuhan, China +1
TrigReason: Trigger-Based Collaboration between Small and Large Reasoning Models
Apr 16, 2026Yi Zhao, Yajuan Peng, Cam-Tu Nguyen +4Efficient Language Model Reasoning LLM Inference Acceleration
AGI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Shanghai Key Laboratory for Intelligent Information Processing, Fudan University · State Key Laboratory for Novel Software Technology, Nanjing University +2
ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving
Apr 16, 2026Yuseon Choi, Jingu Lee, Jungjun Oh +5LLM Serving Mixture-of-Experts Inference
KAIST · Samsung Electronics · GIST
ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding
Apr 16, 2026Walaa Amer, Uday das, Fadi KurdahiSpeculative Decoding LLM Inference Acceleration
Department of Electrical Engineering and Computer Sciences, University of California Irvine, Irvine, USA · Advanced Micro Devices, San Jose, USA
Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference
Apr 15, 2026Xuwen Zhou, Fangxin Liu, Chao Wang +5Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Alibaba Cloud Computing
YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference
Apr 15, 2026You Wu, Ziheng Chen, Yizhen Zhang +6LLM Inference Acceleration KV-Cache Compression
ShanghaiTech University · Alibaba Group · Zhejiang University
ART: Attention Run-time Termination for Efficient Large Language Model Decoding
Apr 15, 2026Chen Qiu, Guozhong Li, Cristian McGee +2Long-Context Language Model Inference LLM Inference Acceleration
King Abdullah University of Science and Technology · University of Central Florida
ToolSpec: Accelerating Tool Calling via Schema-Aware and Retrieval-Augmented Speculative Decoding
Apr 15, 2026Heming Xia, Yongqi Li, Cunxiao Du +2LLM Inference Acceleration LLM Tool Use
Department of Computing, The Hong Kong Polytechnic University · Peking University
Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel
Apr 14, 2026Hongyi Jin, Bohan Hou, Guanjie Wang +18GPU Kernel Optimization GPU Kernel Generation
Carnegie Mellon University · Shanghai Jiao Tong University · NVIDIA +4

Apr 20, 2026J/K move · Enter open · S save

AQPIM: Breaking the PIM Capacity Wall for LLMs with In-Memory Activation Quantization

Kosuke Matsushima, Yasuyuki Okoshi, Masato Motomura, Daichi Fujiki

Institute of Science Tokyo†

Compute-in-Memory Long-Context Language Model Inference Vector Quantization LLM Inference Acceleration Activation Quantization KV-Cache Quantization

Open paper AskPDF arXiv

Processing-in-Memory (PIM) architectures offer a promising solution to the memory bottlenecks in data-intensive machine learning, yet often overlook the growing challenge of activation memory footprint. Conventional PIM approaches struggle with massive KV cache sizes generated in long-context scenarios by Transformer-based models, frequently exceeding PIM's limited memory capacity, while techniques like sparse attention can conflict with PIM's need for data locality. Existing PIM approaches and quantization methods are often insufficient or poorly suited for leveraging the unique characteristics of activations. This work identifies an opportunity for PIM-specialized activation quantization to enhance bandwidth and compute efficiency. We explore clustering-based vector quantization approaches, which align well with activation characteristics and PIM's internal bandwidth capabilities. Building on this, we introduce AQPIM, a novel PIM-aware activation quantization framework based on Product Quantization (PQ), optimizing it for modern Large Language Models (LLMs). By performing quantization directly within memory, AQPIM leverages PIM's high internal bandwidth and enables direct computation on compressed data, significantly reducing both memory footprint and computational overhead for attention computation. AQPIM addresses PQ's accuracy challenges by introducing several algorithmic optimizations. Evaluations demonstrate that AQPIM achieves significant performance improvements, drastically reducing of GPU-CPU communication that can account for 90

\sim

98.5% of decoding latency, together with 3.4

\times

speedup over a SOTA PIM approach.