LLM Inference Acceleration

LLM: Large Language Model

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

Lever: Speculative LLM Inference on Smartphones
May 16, 2026Tuowei Wang, Fengzu Li, Yanfan Sun +2On-Device Language Model Inference LLM Inference
Tsinghua University · Beijing, China · Beihang University +1
Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation
May 15, 2026Shuaiyi Li, Zhisong Zhang, Yan Wang +5Long-Context Modeling LLM Inference Acceleration
The Chinese University of Hong Kong · City University of Hong Kong · Tencent +2
IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression
May 15, 2026Ali Abbasi, Chayne Thrash, Haoran Qin +2LLM Compression Low-Rank Matrix Decomposition
Vanderbilt University · University of California, Davis
STS: Efficient Sparse Attention with Speculative Token Sparsity
May 15, 2026Ceyu Xu, Jiangnan Yu, Yongji Wu +1Self-Attention Long-Context Language Model Inference
The Hong Kong University of Science and Technology · UC Berkeley
Dual-Dimensional Consistency: Balancing Budget and Quality in Adaptive Inference-Time Scaling
May 14, 2026Rongman Xu, Yifei Li, Tianzhe Zhao +3Inference-Time Scaling LLM Inference Acceleration
Xi’an Jiaotong University
GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding
May 14, 2026Fanxu Meng, Zhixin PanGrouped-Query Attention GPU Acceleration
Institute for Artificial Intelligence, Peking University
Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing
May 14, 2026Jie Jiang, Xing Sun, Ruotian Chen +2Speculative Decoding LLM Inference Acceleration
XFP: Quality-Targeted Adaptive Codebook Quantization with Sparse Outlier Separation for LLM Inference
May 14, 2026Thomas WittLLM Quantization Vector Quantization
Gemini Stiftung Leipzig
BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE
May 14, 2026Juntong Wu, Jialiang Cheng, Qishen Yin +5Sparse Mixture-of-Experts Mixture-of-Experts Inference
Taobao & Tmall Group of Alibaba · Shenzhen Graduate School, Peking University
PreFT: Prefill-only finetuning for efficient inference
May 14, 2026Andrew Lanpouthakoun, Aryaman Arora, Zhengxuan Wu +4Fine-Tuning LLM Serving
Stanford University · Tilde Research
Know When To Fold 'Em: Token-Efficient LLM Synthetic Data Generation via Multi-Stage In-Flight Rejection
May 13, 2026Anjir Ahmed Chowdhury, Syed Zawad, Feng YanSynthetic Data Generation LLM Inference Acceleration
Department of Computer Science University of Houston · IBM Research
Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
May 13, 2026Gergely Szilvasy, Manuel Faysse, Maria Lomeli +5Self-Attention KV Caching
Meta FAIR · MICS, CentraleSupélec
Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding
May 13, 2026Shuoyang Sun, Chang Dai, Hao Fang +6Adversarial Attacks on LLMs Speculative Decoding
Harbin Institute of Technology, Shenzhen · South China University of Technology · Tsinghua Shenzhen International Graduate School, Tsinghua University +1
MinT: Managed Infrastructure for Training and Serving Millions of LLMs
May 13, 2026Mind Lab, :, Song Cao +60LLM Serving Adapter Tuning
Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference
May 13, 2026Lingchao Zheng, Yuwei Fan, Jun Li +5LLM Quantization LLM Inference Acceleration
Huawei
N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation
May 13, 2026Aleksander Lorenc, Frédéric Berdoz, Joël Mathys +1Efficient Transformer Inference LLM Inference Acceleration
ETH Zurich
STOP: Structured On-Policy Pruning of Long-Form Reasoning in Low-Data Regimes
May 13, 2026Chenjun Xu, Zhennan Zhou, Zhan Su +3LLM Pruning LLM Fine-Tuning
University of Washington · University of Montreal
Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion
May 12, 2026Chien Van Nguyen, Chaitra Hegde, Van Cuong Pham +3Language Model Decoding Diffusion Language Model Inference
University of Oregon · Google DeepMind · Adobe Research
Search Your Block Floating Point Scales!
May 12, 2026Tanmaey Gupta, Hayden Prairie, Xiaoxia Wu +104-Bit Quantization LLM Inference Acceleration
Cornell University, Ithaca, NY, USA · Together AI, USA · University of California San Diego, La Jolla, CA, USA +1
D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting
May 12, 2026Tianyu Wu, Yu Yao, Zhenting Qi +7Speculative Decoding LLM Inference Acceleration
1Harvard · 2MIT
Efficient LLM-based Advertising via Model Compression and Parallel Verification
May 12, 2026Wenxin Dong, Chang Gao, Guanghui Yu +9LLM Quantization LLM Compression
Baidu Inc. Beijing, China
Ada-MK: Adaptive MegaKernel Optimization via Automated DAG-based Search for LLM Inference
May 12, 2026Wenxin Dong, Mingqing Hu, Guanghui Yu +7GPU Kernel Optimization GPU Acceleration
Baidu Inc.
BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion
May 12, 2026Shaobin Zhuang, Yuang Ai, Jiaming Han +7Multi-Token Prediction Autoregressive Language Modeling
Shanghai Jiao Tong University · MMLab, The Chinese University of Hong Kong · Institute of Automation, Chinese Academy of Sciences +1
Fast MoE Inference via Predictive Prefetching and Expert Replication
May 12, 2026Ankit Jyothish, Ali Jannesari, Aishwarya Sarkar +1Expert Load Balancing GPU Acceleration
Iowa State University Ames, USA
Block-Based Double Decoders
May 11, 2026Asher Labovich, Benjamin Bradley, Vanessa Alexander +1Efficient Transformer Inference LLM Inference Acceleration
Department of Computer Science Brown University Providence, RI 02912
Rethinking LLMOps for Fraud and AML: Building a Compliance-Grade LLM Serving Stack
May 11, 2026Prathamesh Vasudeo Naik, Naresh Dintakurthi, Yue WangLLM Serving Financial Fraud Detection
CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
May 11, 2026Yuning Han, Yangchenchen Jin, Dylan Zhao +1On-Device Language Model Inference Memory-Efficient Inference
University of Florida Gainesville, FL 32611
ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs
May 11, 2026Chayne Thrash, Ali Abbasi, Soheil KolouriRotation-Based Quantization LLM Inference Acceleration
Department of Computer Science Vanderbilt University
SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding
May 11, 2026Anton Plaksin, Sergei Krutikov, Sergei Skvortsov +1LLM Compression Speculative Decoding
Nebius, Amsterdam, Netherlands.
Breaking the Reward Barrier: Accelerating Tree-of-Thought Reasoning via Speculative Exploration
May 11, 2026Shuzhang Zhong, Haochen Huang, Shengxuan Qiu +3Inference-Time Search Tree of Thoughts
Institute for Artificial Intelligence, Peking University · School of Integrated Circuits, Peking University · 3ByteDance Seed

May 16, 2026J/K move · Enter open · S save

Lever: Speculative LLM Inference on Smartphones

Tuowei Wang, Fengzu Li, Yanfan Sun, Wei Gao+1

Tsinghua University · Beijing, China · Beihang University+1

On-Device Language Model Inference LLM Inference Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly needed for interactive mobile applications, but high-quality models exceed the limited DRAM available on smartphones. Flash storage can hold larger models, yet flash-backed inference is slow because autoregressive decoding repeatedly invokes the target model and incurs costly I/O. We observe that speculative decoding is a natural fit for this setting: a small draft model can remain in DRAM, while a larger flash-resident target model verifies multiple candidate tokens per invocation. However, existing methods assume server-class accelerators and fail to account for prolonged I/O latency, limited computation parallelism, and irregular speculation execution. We present Lever, an end-to-end system for efficient flash-backed LLM inference on smartphones. Lever jointly optimizes the three stages of speculative decoding under mobile constraints. For drafting, it builds token trees using an I/O- and compute-aware gain-cost objective. For verification, it prunes low-value branches through early-exit prediction to reduce target-model computation. For execution, it maps speculation efficiently across mobile CPU-NPU hardware to improve utilization. Comprehensive evaluations show that Lever reduces inference latency by an average of 2.93x over baseline flash-offloaded inference and 1.50x over conventional speculative decoding, narrowing the latency gap between flash-backed and memory-resident LLM inference.