LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

Acceptance-Aware Draft Model Training for Speculative Decoding
Sep 21, 2026Tianhua Xia, Mugilan Ganesan, Yifei Feng +3Speculative Decoding LLM Inference Acceleration
New York University · University of Waterloo · Independent Researcher
LumoTree: Path-Parallel Speculative Verification for Hybrid Language Models
Sep 20, 2026Zhiyuan MaSpeculative Decoding LLM Inference Acceleration
On the Efficiency-Safety Dilemma in Large Reasoning Models
Sep 20, 2026Yifei Yang, Zouying Cao, Xingrui Wang +4LLM Inference Acceleration LLM Jailbreak Attacks
AGI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University · Ant Group +2
On-Demand Attention: Language Models Know When to Recall
Sep 17, 2026Haibo Feng, Ruiqi Liang, Hanyang Peng +1Language Model Decoding Long-Context Language Model Inference
Southern University of Science and Technology · Peking University · Peng Cheng Laboratory
To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signals
Sep 17, 2026Roy Eisenstadt, Ido Cohen, Edo Cohen-Karlik +2Speculative Decoding LLM Inference Acceleration
Tel Aviv University · Stealth Startup, Tel Aviv
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
Sep 17, 2026DeepSeek-AI, :, Anyi Xu +590KV Caching Long-Context Language Model Inference
AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines
Sep 16, 2026Li ChenGPU Kernel Optimization Benchmark Design
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Sep 16, 2026Yu Lin, Yiming Wang, Runyuan Cai +2Expert Offloading Mixture-of-Experts Inference
AutoArk
A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality
Sep 16, 2026Jerry KaplanLLM Quantization LLM Evaluation
Adjunct Lecturer, Computer Science and Masters in International Policy, Stanford University
ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
Sep 16, 2026Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao +2Long-Context Language Model Inference LLM Inference Scheduling
University of Southern California, Los Angeles, CA, USA
Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
Sep 15, 2026Vivek KalyanaranganKV-Cache Offloading KV Caching
ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding
Sep 15, 2026Ziyang Ma, Zihong Zhang, Zuchao Li +4Speculative Decoding LLM Inference Acceleration
School of Computer Science, Wuhan University, Wuhan, China · School of Artificial Intelligence, Wuhan University, Wuhan, China · 3Xiaomi, Beijing, China
GrowMTP: Can RL Grow Its Own Draft Head?
Sep 15, 2026Minghua He, Lingzhe Zhang, Yuan Liu +2Speculative Decoding LLM Inference Acceleration
WeChat AI, Tencent · Peking University
Breaking the 1.58-bit Barrier for Ternary LLMs
Sep 14, 2026Evangelos Georganas, Alexander Heinecke, Pradeep DubeyTernary Quantization LLM Compression
Intel Corporation
LLM Inference in a Flash!
Sep 14, 2026Sebastian Zhao, Minseo Kim, Coleman Hooper +5LLM Quantization Compute-in-Memory
UC Berkeley · ICSI · LBNL
To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs
Sep 14, 2026Franck Signe, Hippolyte Pilchen, François Yvon +1Multilingual Language Models LLM Compression
Kyutai, Paris · Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, Grenoble · Sorbonne Université, CNRS, Institute of Intelligent Systems and Robotics, Paris
Evaluating Losslessness in Speculative Decoding Under Finite-Precision Inference
Sep 14, 2026Ilya Koziev, Leonid Sinev, Ivan OseledetsSpeculative Decoding LLM Inference Acceleration
T-LoopFormer: Token-Level Elastic-Depth Looped Transformers for Latent Reasoning With Dynamic Routing
Sep 14, 2026Mingqian Yu, Wenpeng Zhang, Peilin ZhaoKV Caching Recurrent Transformers
Institute of Automation Chinese Academy of Sciences Beijing, China · Independent Reasearcher · School of Artificial Intelligence Shanghai Jiao Tong University Shanghai, China
SpliTEE: Fast and Private LLM Inference by Coupling GPU-Assisted Trusted Execution Environments with Differential Privacy
Sep 14, 2026Shashie Dilhara Batan Arachchige, Robin Carpentier, Hassan Jameel Asghar +1Privacy-Preserving Language Models Split Inference
School of Computing, Macquarie University
Validating Hybrid-State Cache Recovery for GLM-5.3-Flash with vLLM and LMCache
Sep 14, 2026Frank LiLLM Inference Efficiency KV-Cache Management
UNSW Sydney
Shared KV Caching for Replicated 27B Inference: Correctness Failures and Performance Boundaries
Sep 14, 2026Frank LiKV Caching KV-Cache Management
UNSW Sydney
RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems
Sep 14, 2026Ziyue Yang, Yuting Jiang, Lei Qu +1LLM Inference Large Language Model-Guided Optimization
Shanghai Xingyunzhili Artificial Intelligence Institute, Shanghai, China · Microsoft Research, Redmond, WA, USA
Rethinking Heterogeneous System Disaggregation for Subquadratic Attention
Sep 14, 2026Arya Tschand, Yaosheng Fu, Vikram Sharma Mailthody +6Disaggregated LLM Serving LLM Inference Acceleration
Harvard University Cambridge, Massachusetts, USA · NVIDIA Santa Clara, California, USA
AgentKV: Phase-Aware KV Eviction for Agentic LLMs
Sep 14, 2026Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao +3KV Caching KV-Cache Eviction
Imperial College London, London, United Kingdom · Columbia University, New York, NY, USA
Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference
Sep 13, 2026Tian JinLong-Context Language Model Inference Efficient Language Model Reasoning
Department of Electrical Engineering and Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY
FlexComp: One Model for Every Ratio in Context Compression
Sep 11, 2026Kaiyan Zhao, Zhongtao Miao, Akiko Aizawa +1LLM Inference Acceleration Context Compression
The University of Tokyo · National Institute of Informatics
OmniKVQuant: KV Cache Quantization for Omni-LLMs
Sep 11, 2026Suho Yoo, Hyunjong Ok, Jongmin Choi +2Efficient Multimodal Inference KV Caching
KAIST · POSTECH
Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
Sep 11, 2026Joseph Kanichai, Tiziano De Matteis, Animesh TrivediKV-Cache Offloading KV Caching
Vrije Universiteit Amsterdam Amsterdam, the Netherlands · IBM Research Zurich Zurich, Switzerland
A Voice-Interactive Multi-Agent System for Smart Operating Rooms: Architecture Design and Key Technologies
Sep 10, 2026Tianxiang ZhouLanguage Model-Based Control LLM Agent Orchestration
Wuhan United Imaging Surgical Co., Ltd. (UIS) Wuhan, China
Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts
Sep 10, 2026Dohyeon Kim, Bedionita Soro, Sung Ju HwangParameter-Free Mixture-of-Experts Routing Sparse Mixture-of-Experts
KAIST

Sep 21, 2026, cs.LGJ/K move · Enter open · S save

Acceptance-Aware Draft Model Training for Speculative Decoding

Tianhua Xia, Mugilan Ganesan, Yifei Feng, Haiyu Wang+2

New York University · University of Waterloo · Independent Researcher

Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to generate multiple candidate tokens that are verified by the target model in a single forward pass. Its speedup is largely determined by the acceptance length, yet existing draft-model training methods mainly optimize cross-entropy or Kullback-Leibler (KL) divergence as proxies. These objectives encourage distribution matching but do not directly optimize acceptance length, and the acceptance mechanism also differs between greedy and sampling-based decoding. In this work, we propose acceptance-length-aware training losses that directly optimize the expected number of accepted tokens within a speculative window. For greedy verification, we derive an expected accepted length (EAL) loss that explicitly maximizes expected acceptance length. For sampling-based decoding, we introduce a window total variation (WTV) loss that optimizes the overlap between temperature-scaled draft and target distributions while accounting for sequential acceptance dependencies. Both objectives can be further combined with a group-relative reinforcement learning stage (GRPO) using simulated acceptance length as the reward. Experiments across different target and draft models, tasks, and decoding settings show that our losses consistently improve acceptance length over KL-based training. WTV provides particularly strong gains under sampling-based decoding, while EAL better matches greedy verification. These results show that directly optimizing the acceptance objective, with losses tailored to the decoding mode, is more effective than conventional distribution-matching objectives.

Momentum

Related topics

Latest papers 677