LLM Inference Acceleration

LLM: Large Language Model

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility
May 7, 2026Jungsuk Oh, Hyeseo Jeon, Hyunjune Ji +2Self-Attention Long-Context Language Model Inference
Graduate School of Data Science Seoul National University Seoul, Republic of Korea
Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inference
May 7, 2026Saksham Rathi, Preeti, Mythili VutukuruLLM Inference Scheduling KV-Cache Management
Indian Institute of Technology Bombay, Mumbai, India
HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devices
May 7, 2026Shen Xu, Xiangwen Zhuge, Zhe Xu +3LLM Compression Memory-Efficient Inference
Tsinghua University · Huazhong University of Science and Technology
When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon
May 7, 2026Mohamed Amine BergachGPU Kernel Optimization KV Caching
Illumina, San Diego, CA, USA
Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving
May 7, 2026Bole Ma, Jan Eitzinger, Harald KöstlerKV Caching LLM Inference Serving
Erlangen National High Performance Computing Center · Erlangen, Germany
UniVer: A Unified Perspective for Multi-step and Multi-draft Speculative Decoding
May 6, 2026Yepeng Weng, Qiao Hu, Takehisa YairiSpeculative Decoding LLM Inference Acceleration
The University of Tokyo · 2Lenovo AI Technology Center · National Center for Mathematics and Interdisciplinary Sciences (NCMIS), AMSS, CAS
Budgeted LoRA: Distillation as Structured Compute Allocation for Efficient Inference
May 5, 2026Mohammed Sabry, Anya BelzLLM Compression Low-Rank Adaptation
ADAPT Centre, Dublin City University, Ireland
Parallel Prefix Verification for Speculative Generation
May 5, 2026Yuncheng Yao, Yuxuan Xia, Shengjie Wang +1Speculative Decoding LLM Inference Acceleration
Duke University · New York University
AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse
May 5, 2026Jie Ou, Jinyu Guo, Shiyao Guo +5Long-Context Language Model Inference KV-Cache Management
School of Information and Software Engineering, University of Electronic Science and Technology of China
Performance Optimization and Comparative Analysis of Generative AI Models on Advanced Accelerators
May 5, 2026Amitash Nanda, Javier Hernandez Nicolau, Madhusudan Gujral +3Mixed-Precision Quantization AI Accelerator Inference
University of California San Diego, San Diego, CA, USA · San Diego Supercomputer Center, San Diego, CA, USA
SchemaRAG: Dynamic Large Schema Reduction for LLM-driven Structured Information Extraction
May 4, 2026Sin Yu Bonnie Ho, Arlie Coles, Erik Larsson +3Retrieval-Augmented Generation LLM Inference Acceleration
Microsoft
Gated Subspace Inference for Transformer Acceleration
May 4, 2026Stephen J. ThomasTransformer Inference LLM Inference Acceleration
Department of Computer Science and Engineering, Lehigh University, Bethlehem, PA 18015
SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection
May 4, 2026Shikhar ShuklaSpeculative Decoding LLM Inference Acceleration
University of Kentucky, Lexington, KY, USA
The Model Knows, the Decoder Finds: Future Value Guided Particle Power Sampling
May 4, 2026Tu Nguyen, Matthieu Zimmer, Rasul Tutunov +2Language Model Decoding LLM Inference Acceleration
1Huawei Heisenberg Research Center · 2Huawei Noah’s Ark Lab · 3UCL Centre for Artificial Intelligence
Statistically-Lossless Quantization of Large Language Models
May 4, 2026Michael Helcig, Eldar Kurtic, Dan AlistarhLLM Quantization LLM Inference Acceleration
ETH Z̈urich · ISTA
HALO: Hybrid Adaptive Latent Reasoning for Language Models
May 3, 2026Micah ZhangLLM Inference Acceleration Adaptive Computation
Lockheed Martin
Stochastic Sparse Attention for Memory-Bound Inference
May 3, 2026Kyle Lee, Corentin Delacour, Kevin Callahan-Coray +5GPU Acceleration Self-Attention
Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA, USA · Department of Electrical and Computer Engineering, University of Michigan, Ann Arbor, MI, USA · Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, USA
Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism
May 3, 2026Long Zhao, Qinghe Wang, Jiaan Zhu +5Tensor Parallelism LLM Inference Acceleration
Anhui University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · University of Science and Technology of China +2
SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
May 3, 2026Yipin Guo, Siddharth JoshiDisaggregated LLM Serving KV-Cache Compression
University of Notre Dame IN, USA
KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
May 1, 2026Shen Han, Yuyang Wu, Junpu Yu +1KV Caching Memory-Efficient Inference
Carnegie Mellon University
Tempus: A Temporally Scalable Resource-Invariant GEMM Streaming Framework for Versal AI Edge
May 1, 2026M. Grailoo, J. Núñez-YáñezLLM Inference Acceleration FPGA Acceleration
∗Linköping University, Sweden · †Universidad Politecnica de Madrid, Spain
BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs
May 1, 2026Zhixiong Zhao, Zukang Xu, Dawei YangLLM Quantization LLM Inference Acceleration
1Houmo AI
Rethinking LLM Ensembling from the Perspective of Mixture Models
May 1, 2026Jiale Fu, Yuchu Jiang, Peijun Wu +3LLM Inference Acceleration LLM Routing
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education · Southeast University · Qiyuan Tech +2
Caracal: Causal Architecture via Spectral Mixing
Apr 30, 2026Bingzheng Gan, Tianyi Zhang, Yusu Li +4Long-Context Language Modeling Autoregressive Language Modeling
Huawei Technologies Co., Ltd., China.
Position-Aware Drafting for Inference Acceleration in LLM-Based Generative List-Wise Recommendation
Apr 30, 2026Jiaju Chen, Chongming Gao, Chenxiao Fan +4Speculative Decoding LLM Inference Acceleration
University of Science and Technology of China, Hefei, China · Independent Researcher
To Diff or Not to Diff? Structure-Aware and Adaptive Output Formats for Efficient LLM-based Code Editing
Apr 30, 2026Wei Cheng, Yongchang Cao, Chen Shen +4Code Generation LLM Inference Acceleration
State Key Laboratory for Novel Software Technology, Nanjing University, China · Tongyi Lab, Alibaba Group, China
Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding
Apr 29, 2026Hayate Iso, Tiyasa Mitra, Sudipta Mondal +15Reinforcement Learning Speculative Decoding
NVIDIA
When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?
Apr 29, 2026Tianyu Liu, Yuhao Shen, Xinyi Hu +8Speculative Decoding LLM Inference Acceleration
1Qwen Applications Business Group of Alibaba · University of Science and Technology of China · 3Zhejiang University
Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens
Apr 29, 2026Zhenyu Zhao, Sander Land, Daniel M. Bikel +1LLM Compression LLM Interpretability
Writer, Inc.
Efficient, VRAM-Constrained xLM Inference on Clients
Apr 29, 2026Aditya Ukarande, Deep Shekhar, Marc Blackstein +1GPU Acceleration Efficient VLM Inference
NVIDIA Corporation, Santa Clara, CA, USA · NVIDIA Graphics Pvt Ltd., Bangalore, India

May 7, 2026J/K move · Enter open · S save

Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility

Jungsuk Oh, Hyeseo Jeon, Hyunjune Ji, Kyongmin Kong+1

Graduate School of Data Science Seoul National University Seoul, Republic of Korea

Self-Attention Long-Context Language Model Inference KV-Cache Management LLM Inference Acceleration

Open paper AskPDF arXiv

Long-context inference in decoder-only language models is costly because long prompts are processed during Prefill, cached at every layer, and repeatedly attended to during autoregressive Decode. We introduce Shallow Prefill, dEEp Decode (SPEED), a phase-asymmetric KV-visibility policy that materializes non-anchor prompt-token KV states only in lower layers while keeping Decode-phase tokens full-depth. Unlike previous approaches that make upper-layer prompt KV states cheaper to store or construct, SPEED removes prefill tokens from the upper-layer Decode visibility set altogether. With a minimal BoS anchor, this simple change preserves broad benchmark quality while reducing long-context cost. In a controlled Llama-3.1-8B instruction-tuning study, SPEED using only 75% of layers for prefill tokens reaches 51.2 average score on OLMES-style benchmarks, compared with 51.4 for the full-depth baseline, while improving TTFT by 33%, TPOT by 22%, and reducing active KV memory by 25.0% at 128K context. Layer-wise diagnostics suggest that this cutoff retains the main prompt-selection and representation-stabilization regions of the full-depth model. These results show that long-context prompt tokens need not always persist as full-depth KV-cache objects when Decode-phase tokens remain full-depth.