LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
Jul 12, 2026Zipeng Gao, Zhi Zheng, Qingrong Xia +5Speculative Decoding LLM Inference Acceleration
1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Huawei Technologies Co., Ltd.
MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference
Jul 12, 2026Venkatesha Matam, Keon KimKV Caching KV-Cache Eviction
Independent Researcher New York City, USA
Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices
Jul 11, 2026Yangyijian Liu, Hongyi Ye, Mingyang Li +1On-Device Language Model Inference LLM Inference Scheduling
School of Computer Science, Nanjing University Nanjing, China
COBS: Cumulant Order Block Sparse Attention
Jul 10, 2026Alexander Tian, Aditya Ghai, Sanjit Neelam +2KV Caching LLM Inference Acceleration
MatX
Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models
Jul 9, 2026Bishmoy Paul, Youngmin Yi, Hoeseok YangLLM Pruning Activation Sparsity
Santa Clara University · Sogang University
BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Serving
Jul 9, 2026Yuanjie Zhu, Liangwei Yang, Ke Xu +4LLM Inference Scheduling Diffusion Model Serving
University of Illinois Chicago, USA
A Practical Investigation of Training-free Relaxed Speculative Decoding
Jul 9, 2026Guoxuan Xia, Luka Ribar, Paul BalancaSpeculative Decoding LLM Inference Acceleration
Graphcore
DominoTree: Conditional Tree-Structured Drafting with Domino for Speculative Decoding
Jul 9, 2026Saw S. Lin, Jyh-Shing Roger JangSpeculative Decoding LLM Inference Acceleration
Zhiqi Zhang · Department of Computer Science and Information Engineering, National Taiwan University.
Prompt Compression via Activation Aggregation
Jul 9, 2026Thibaud Ardoin, Semira Einsele, Evis Bregu +1LLM Inference Acceleration Representation Geometry in Language Models
Freie Universität Berlin
Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention
Jul 9, 2026Ryota Kobayashi, Tsubasa Hirakawa, Takayoshi Yamashita +4LLM Pruning LLM Compression
1Chubu University · 2Panasonic Holdings Corporation
CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems
Jul 8, 2026Tingkai Liu, Muralidhar Andoorveedu, Sanjoy Das +2GPU Kernel Optimization High-Performance Computing
University of Illinois Urbana-Champaign · NVIDIA Corporation
The Key to Going Linear: Analysis-Driven Transformer Linearization
Jul 8, 2026Anna Kuzina, Paul N. Whatmough, Babak Ehteshami BejnordiSoftmax Attention LLM Inference Acceleration
Qualcomm AI Research∗ · Qualcomm AI Research
DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting
Jul 8, 2026Hong-Kai Zheng, Piji LiSpeculative Decoding LLM Inference Acceleration
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China · The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China
TF-Engram: A Train-Free Engram with SSD-Backed Memory for Large Language Models
Jul 8, 2026Yutang Ma, Kecheng Huang, Xikun Jiang +1Memory-Augmented Language Models Knowledge Augmentation for Language Models
The Chinese University of Hong Kong · Beijing Institute of Technology, Zhuhai
Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE
Jul 8, 2026Haozhan Tang, Zerui Wang, Yuxian Gu +2Rotary Positional Embeddings Long-Context Language Modeling
NVIDIA
Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs
Jul 7, 2026Ke-Han Lu, Keqi Deng, Ruchao Fan +2KV Caching Automatic Speech Recognition
Microsoft, USA · National Taiwan University, Taiwan
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
Jul 7, 2026Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo +4KV Caching Long-Context Language Model Inference
Instituto de Investigación en Visión Artificial
FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
Jul 7, 2026Anna Córdoba, Adam Puente Tercero, Nerea Angulo Hijo +4KV Caching Long-Context Language Model Inference
Instituto de Investigación en Visión Artificial
DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
Jul 6, 2026Xin Cheng, Xingkai Yu, Chenze Shao +30LLM Serving Speculative Decoding
1Peking University · 2DeepSeek-AI
KVpop -- Key-Value Cache Compression with Predictive Online Pruning
Jul 6, 2026Lukas Hauzenberger, Niklas Schmidinger, Anamaria-Roberta Hartl +5KV Caching KV-Cache Eviction
1NXAI · 2Johannes Kepler University Linz, Austria
Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B
Jul 5, 2026Jaeyeon Kim, Jewon Lee, Bo-Kyeong KimLLM Quantization Speculative Decoding
Nota Inc.
Uncertainty-gated selection for block-sparse attention
Jul 4, 2026Thomas RossiLong-Context Language Model Inference LLM Inference Acceleration
Eonpass
ELiTeFormer: An Efficient Transformer for FPGAs
Jul 4, 2026Victor Agostinelli, Nicolas Bohm Agostini, Antonino TumeoLLM Quantization Ternary Quantization
Oregon State University Corvallis, Oregon, USA · Pacific Northwest National Laboratory Richland, Washington, USA
Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models
Jul 3, 2026Hamish OgilvyMixed-Precision Quantization LLM Quantization
Algolia Research
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
Jul 2, 2026Shrikara Arun, Anjaly Parayil, Srikant Bharadwaj +2LLM Serving Disaggregated LLM Serving
Microsoft
Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference
Jul 2, 2026Wenchen Han, Gingfung Matthew Yeung, Marco Barletta +3LLM Inference KV Caching
University College London · Huawei
PARTREP: Learning What to Repeat for Decoder-only LLMs
Jul 2, 2026Andikawati P Widjaja, Yongjun Kim, Hyounghun Kim +1Decoder-Only Language Models LLM Prompting
♡Bandung Institute of Technology · ♠Pohang University of Science and Technology
Message Passing Enables Efficient Reasoning
Jul 1, 2026Xuecheng Liu, Daman Arora, Gokul Swamy +1Inference-Time Scaling Efficient Language Model Reasoning
Carnegie Mellon University
MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression
Jul 1, 2026Sheng Qiang, Ruiwei Chen, Yinpeng Wu +5KV Caching Long-Context Language Model Inference
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University, Shanghai, China
BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal
Jul 1, 2026Prabod Rathnayaka, Fabian Waschkowski, Lukas WesemannAI Accelerator Inference On-Device Language Model Inference
Base Compute, Melbourne, Australia

Jul 12, 2026, cs.CLJ/K move · Enter open · S save

Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting

Zipeng Gao, Zhi Zheng, Qingrong Xia, Junda Lin+4

1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Huawei Technologies Co., Ltd.

Speculative Decoding LLM Inference Acceleration Autoregressive Decoding

Open paper AskPDF arXiv

Speculative decoding has significantly accelerated Large Language Model (LLM) inference by alleviating memory-bound bottlenecks. However, traditional speculative decoding typically relies on auxiliary draft modules, incurring significant training and communication overhead. Although recent methods attempt to generate drafts within the target model itself, they often fail to fully exploit its latent parallel capacity due to a lack of structural coordination. In this paper, we propose \textbf{Progressive Tree Drafting (PTD)}, which employs a structured, guided parallel drafting strategy to harness the model's parallel potential. By coupling a progressive tree structure with a stepwise pruning mechanism, PTD actively guides the LLM to explore multiple semantic paths in a single forward pass, ensuring both draft diversity and coherence. Experiments demonstrate that PTD achieves up to

2\times

decoding speedup across various benchmarks while remaining training-free and model-agnostic. Our code is available at: https://github.com/MINE-USTC/PTD.

Momentum

Related topics

Latest papers 677