LLM Inference Acceleration

LLM: Large Language Model

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

GELATO: Generative Entropy- and Lyapunov-based Adaptive Token Offloading for Device-Edge Speculative LLM Inference
May 11, 2026Zengzipeng Tang, Yuxuan Sun, Wei Chen +2On-Device Language Model Inference Edge Computing
School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China
Attention Drift: What Autoregressive Speculative Decoding Models Learn
May 11, 2026Doğaç Eldenk, Payal Mohapatra, Yigitcan Comlek +3Long-Context Language Model Inference Speculative Decoding
Northwestern University · GE Aerospace · fal +1
Merlin: Deterministic Byte-Exact Deduplication for Lossless Context Optimization in Large Language Model Inference
May 11, 2026Sietse SchelpeLLM Inference Acceleration
Corbenic AI, Inc.
Entropy-informed Decoding: Adaptive Information-Driven Branching
May 10, 2026Benjamin Patrick Evans, Sumitra Ganesh, Leo ArdonLanguage Model Decoding LLM Inference Acceleration
JP Morgan AI Research, London, UK · JP Morgan AI Research, New York, USA
KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving
May 10, 2026Zhiqing Zhong, Zhijing Ye, Jian Zhang +3KV Caching KV-Cache Management
Stevens Institute of Technology Hoboken, New Jersey, USA · The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China · Argonne National Laboratory Lemont, Illinois, USA +1
Test-Time Speculation
May 10, 2026Avinash Kumar, Sujay Sanghavi, Poulami DasTest-Time Adaptation Test-Time Optimization
The University of Texas at Austin
Sparse Layers are Critical to Scaling Looped Language Models
May 9, 2026Ryan Lee, Jacob Biloki, Edward J. Hu +1Mixture-of-Experts Language Models Recurrent Transformers
1USC Information Sciences Institute · 2Netflix · 3Independent Researcher
ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference
May 9, 2026Junjie Li, Jiong Lou, Jie LiLLM Pruning KV Caching
Shanghai Jiao Tong University
BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learning
May 9, 2026Yuhang Xu, Kaibin Tian, Yang Tian +6Reinforcement Learning Speculative Decoding
1Shanghai Jiao Tong University · 2Bytedance
ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing
May 9, 2026Yongqi An, Chang Lu, Kuan Zhu +5KV Caching Long-Context Language Model Inference
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · University of Electronic Science and Technology of China, Chengdu, China +2
LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss
May 9, 2026Euntae Choi, Sumin Song, Sungjoo YooLLM Quantization Quantization-Aware Training
Seoul National University
Structured Recurrent Mixers for Massively Parallelized Sequence Generation
May 9, 2026Benjamin L. BadgerRecurrent Neural Networks Autoregressive Language Modeling
IBM
PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding
May 9, 2026Zihao An, Taichi Liu, Ziqiong Liu +3Speculative Decoding LLM Inference Acceleration
Advanced Micro Devices, Inc. · Rutgers University · Michigan State University
HAServe: Horizon-Aware Scheduling and Memory Co-Design for Efficient LLM Serving
May 9, 2026Xingyu Qu, Tianhao Lin, Yiqi Li +2LLM Serving KV Caching
School of Computer Science, Wuhan University · Amazon
Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
May 9, 2026Jongseok Park, Sunga Kim, Zhenyu Gu +2Activation Sparsity Mixture-of-Experts Inference
University of California, Berkeley · Work done during an internship at Advanced Micro Devices, Inc. · Advanced Micro Devices, Inc.
Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compression
May 9, 2026Hengyi Zhu, Zhendong Mi, Grace Li Zhang +1LLM Compression Low-Rank Matrix Decomposition
1Stevens Institute of Technology, Hoboken, USA · 2Technical University of Darmstadt, Darmstadt, Germany
Fast Byte Latent Transformer
May 8, 2026Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz +5Speculative Decoding LLM Inference Acceleration
FAIR at Meta · Stanford University · University of Washington
FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast
May 8, 2026Wenhao Wu, Zishan Shao, Kangning Cui +5Efficient Transformer Inference LLM Compression
Department of Electrical & Computer Engineering, Duke University · Department of Statistical Science, Duke University · Department of Computer Science, Wake Forest University +1
An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
May 8, 2026Feiyu Yao, Zhixiong Niu, Xiaqing Li +3GPU Acceleration Self-Attention
Beijing University of Technology Beijing, China · Microsoft Research Beijing, China · Beijing Jiaotong University Beijing, China
Priming: Hybrid State Space Models From Pre-trained Transformers
May 8, 2026Aditya Chattopadhyay, Elvis Nunez, Prannay Kaul +6State Space Models Long-Context Language Model Inference
AWS Agentic AI
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
May 8, 2026Ruijie Zhou, Fanxu Meng, Yufei Xu +4Self-Attention Long-Context Language Model Inference
CSR: Infinite-Horizon Real-Time Policies with Massive Cached State Representations
May 8, 2026Robin Karlsson, Go SuzuiRobotic Control Language Model-Based Control
1GODOT Inc, Hyogo, Japan · 2Graduate School of Informatics, Nagoya University, Aichi, Japan.
LaTER: Efficient Test-Time Reasoning via Latent Exploration and Explicit Verification
May 8, 2026Xuan Li, Yining Wang, Yuchen Liu +7CoT Reasoning Efficient Language Model Reasoning
University of Science and Technology of China · Unisound AI Technology Co., Ltd · MAIS, Institute of Automation, Chinese Academy of Sciences
SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting
May 8, 2026Weijie Shi, Qiang Xu, Fan Deng +9Speculative Decoding LLM Inference Acceleration
Hong Kong University of Science and Technology · MetaX · Zhejiang Normal University +1
Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
May 7, 2026Mohsen Dehghankar, Abolfazl AsudehSelf-Attention KV Caching
Department of Computer Science University of Illinois Chicago
Grokers: Bottom-Up Inductive Comprehension and Write-Time Intelligence over Typed Knowledge Graphs
May 7, 2026Gregory MagarshakKG Construction Memory-Augmented Language Models
Qbix, Inc. & Intercoin, Inc. · IE University NYC · New York, NY, USA
Litespark Inference For CPUs: Ultra-Fast SIMD Framework for Ternary (1.58-bit) Language Models
May 7, 2026Nii Osae Osae Dade, Tony Morri, Moinul Hossain Rahat +2LLM Quantization Efficient Neural Network Inference
Mindbeam AI
LatentRAG: Latent Reasoning and Retrieval for Efficient Agentic RAG
May 7, 2026Yijia Zheng, Marcel WorringLLM Inference Acceleration Agentic RAG
University of Amsterdam, Amsterdam, the Netherlands
UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification
May 7, 2026Qihang Fan, Huaibo Huang, Zhiying Wu +2Long-Context Language Model Inference LLM Inference Acceleration
1MAIS&NLPR, CASIA · 2UCAS · 3WeChat, Tencent
Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
May 7, 2026Muhammad Shahir Abdurrahman, Chun Deng, Azalia Mirhoseini +1Expert Parallelism Mixture-of-Experts Inference
Stanford University

May 11, 2026J/K move · Enter open · S save

GELATO: Generative Entropy- and Lyapunov-based Adaptive Token Offloading for Device-Edge Speculative LLM Inference

Zengzipeng Tang, Yuxuan Sun, Wei Chen, Jianwen Ding+1

School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China

On-Device Language Model Inference Edge Computing Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

The recent growth of on-device Large Language Model (LLM) inference has driven significant interest in device-edge collaborative LLM inference. As a promising architecture, Speculative Decoding (SD) is increasingly adopted where a lightweight draft model rapidly generates candidate tokens to be verified by a powerful target model. However, a fundamental challenge lies in achieving per-token resource scheduling to effectively adapt SD paradigm to resource-constrained edge environment. This paper proposes a Generative Entropy- and Lyapunov-based Adaptive Token Offloading framework, named GELATO, to maximize decoding throughput under energy constraints in a device-edge collaborative SD system. Specifically, an outer drift-plus-penalty loop makes online decisions to establish a reference drafting budget, managing long-term energy-throughput trade-off. Further, a nested entropy-driven generation mechanism executes early exiting to adapt to per-token dynamic generative uncertainty. Theoretical analysis establishes a rigorous performance bound on long-term throughput for GELATO. Extensive evaluations demonstrate that GELATO achieves a globally optimal tradeoff, outperforming state-of-the-art distributed SD architectures by 64.98% in token throughput and reducing energy consumption by 47.47% under resource-constrained environments, while preserving LLM decoding quality.