Autoregressive Decoding

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 38

All topics

Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution
Oct 6, 2026Shuhao Li, Fanghua Ye, Wanyu Lin +2Speculative Decoding LLM Inference Acceleration
EIT-NLP Lab, Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University · Independent Researcher
DLoop: Looped Speculative Decoding
Oct 6, 2026Geonmo Gu, Byeongho Heo, HeeJae Jun +4Speculative Decoding LLM Inference Acceleration
NAVER AI Lab · Korea University · NAVER AI Search Platform
RVQ Position Aware Speculative Decoding for On Device Text to Speech
Sep 29, 2026Berkin Durmus, Eduardo Pacheco, Zach Nagengast +1TTS Synthesis Streaming TTS Synthesis
Argmax, Inc., USA
FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backward
Sep 27, 2026Sriman AchantaSoftmax Attention LLM Inference Acceleration
Virginia Commonwealth University
Adapting Tree-Structured Speculative Decoding to DeepSeek-V4 for Efficient Inference
Sep 21, 2026Changxu Liu, Zhaogeng LiSpeculative Decoding LLM Inference Acceleration
Baige AI Team, Baidu Inc. · Fudan University
One Patch, Three Roles: What Is Actually Coupled in Autoregressive Time-Series Forecasting?
Sep 20, 2026Ziang Li, Yue Huang, Guoxu Zhou +4Long-Term Time Series Forecasting Time Series Forecasting
Guangdong University of Technology · Guangdong Polytechnic Normal University · Harbin Institute of Technology
Evaluating Losslessness in Speculative Decoding Under Finite-Precision Inference
Sep 14, 2026Ilya Koziev, Leonid Sinev, Ivan OseledetsSpeculative Decoding LLM Inference Acceleration
OneLA: Scaling Linear-Attention Decoding to Large Beams in Generative Recommendation
Sep 14, 2026Xiangrui Yang, Cheng Peng, Yunfeng Zhao +9Linear Attention Autoregressive Decoding
Kuaishou Technology China · Peking University China
Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference
Sep 13, 2026Tian JinLong-Context Language Model Inference Efficient Language Model Reasoning
Department of Electrical Engineering and Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY
Line-Coupled Language Model
Sep 7, 2026Shiyuan Li, Shaorong Zhang, Zhaorui Yang +3Multi-Token Prediction Language Model Decoding
University of California, Riverside Riverside, CA, USA
Masked Autoregressive Speech Enhancement with Continuous Neural Audio Codec Representations
Sep 3, 2026Yoto Fujita, Simon Leglaive, Laurent GirinSpeech Enhancement Autoregressive Decoding
CentraleSup´elec, IETR (UMR CNRS 6164), France · Univ. Grenoble Alpes, CNRS, Grenoble-INP, GIPSA-lab, France
GeoPAR: Large-Scale Multi-Agent Combinatorial Optimization with Geometry-Guided Parallel Autoregressive Learning
Sep 1, 2026Wenjian Wu, Zesheng Jia, Jiaying Tang +2Combinatorial Optimization Vehicle Routing
School of Future Science and Engineering, Soochow University, Suzhou, China
ReLMCodec: Designing Predictable Speech Tokens from Pre-Quantization Phoneme Structure
Aug 8, 2026Zixiang Wan, Xusheng Yang, Zheng Wang +1Neural Audio Codecs Audio Tokenization
1Peking University, Shenzhen, China · 2Tencent, Shenzhen, China
Approximate Speculative Decoding
Aug 4, 2026Yuannuo Feng, Zegang Peng, Yuxin Xie +5Speculative Decoding LLM Inference Acceleration
School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Precision Instrument, Tsinghua University, Beijing, China · Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China +2
From Chains to Trees: Parent-Conditioned Drafting for Semi-Autoregressive Speculative Decoding
Aug 3, 2026Zixian Li, Tong Li, Chi Xie +2Speculative Decoding LLM Inference Acceleration
OPPO AI Center
Wavefront Parallelization for Efficient Learned Image Compression
Jul 21, 2026Shimon Murai, Fangzheng Lin, Kasidis Arunruangsirilert +1Learned Image Compression Inference-Time Optimization
School of Fundamental Science and Engineering Waseda University Tokyo, Japan · School of Engineering Institute of Science Tokyo Tokyo, Japan
Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
Jul 12, 2026Zipeng Gao, Zhi Zheng, Qingrong Xia +5Speculative Decoding LLM Inference Acceleration
1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Huawei Technologies Co., Ltd.
Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inference
Jul 10, 2026Junfei Zhan, Haoxun Shen, Mingang Guo +2Efficient VLM Inference Edge Inference
Department of Electrical and Systems Engineering University of Pennsylvania Philadelphia, PA, USA · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences Shenzhen, China · College of Information Science and Technology Jinan University Guangzhou, China
A Practical Investigation of Training-free Relaxed Speculative Decoding
Jul 9, 2026Guoxuan Xia, Luka Ribar, Paul BalancaSpeculative Decoding LLM Inference Acceleration
Graphcore
KVpop -- Key-Value Cache Compression with Predictive Online Pruning
Jul 6, 2026Lukas Hauzenberger, Niklas Schmidinger, Anamaria-Roberta Hartl +5KV Caching KV-Cache Eviction
1NXAI · 2Johannes Kepler University Linz, Austria
Depth Exploration for LLM Decoding
Jun 28, 2026Weisi Yang, Zipeng Sun, Stephen XiaLanguage Model Decoding LLM Inference Acceleration
Northwestern University · MILA – Quebec AI Institute · McGill University
SSD: Spatially Speculative Decoding Accelerates Autoregressive Image Generation
Jun 18, 2026Shilong Xiang, Zirui Zhang, Lijun Yu +1Visual Autoregressive Models Speculative Decoding
Rutgers University
A Generalized Formalism of Auto-Regressive Decoding for Speech Processing
Jun 16, 2026Julia Gachot, Philipp Allgeuer, Marie S. Bauer +1Speech Processing Autoregressive Decoding
Knowledge Technology, Department of Informatics, University of Hamburg, Germany
Real-Time Execution with Autoregressive Policies
Jun 11, 2026Sangkyu Lee, Seohyeon Park, Tackgeun You +4Robotic Control Vision-Language-Action Models
Korea Institute of Science and Technology · Seoul National University · Google Research
MemNovo: Look Back at the Spectrum for Balanced De Novo Peptide Sequencing from Mass Spectrometry
Jun 10, 2026Dongxin Lyu, Jingbo Zhou, Hongxin Xiang +2Memory-Augmented Neural Networks Transformer Inference
Westlake University Hangzhou, Zhejiang, China · Hunan University Changsha, Hunan, China · Shanghai Artificial Intelligence Laboratory Shanghai, China +1
ANCHOR: Autoregressive Non-intrusive Chunk-Ordered Refinement for Joint Multi-Resolution Speech Quality Modeling
Jun 8, 2026Zhuoyan Tao, Jiatong Shi, Hye-jin Shim +1Speech Quality Assessment Autoregressive Decoding
University of Southern California, USA · Carnegie Mellon University, USA
AsyncLane: Decoupling Refinement from Advancement in Diffusion Language Model Decoding
Jun 7, 2026Yingxuan Ren, Yuxuan Lou, Yong Liu +4LLM Inference Scheduling Blockwise Diffusion Language Models
1National University of Singapore · University of Southampton
Parallel Jacobi Decoding for Fast Autoregressive Image Generation
Jun 4, 2026Boya Liao, Ying Li, Siyong Jian +1Visual Autoregressive Models Autoregressive Image Generation
Westlake University
EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture
May 22, 2026Bowen Duan, Cong Guo, Chiyue Wei +8Vector Quantization LLM Inference Acceleration
Duke University, USA
FlashAR: Efficient Post-Training Acceleration for Autoregressive Image Generation
May 10, 2026Junkang Zhou, Yefei He, Feng Chen +2Parallel Decoding Autoregressive Image Generation
Zhejiang University, China · University of Adelaide, Australia

Oct 6, 2026, cs.CLJ/K move · Enter open · S save

Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution

Shuhao Li, Fanghua Ye, Wanyu Lin, Tianyu Yuan+1

EIT-NLP Lab, Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University · Independent Researcher

Speculative Decoding LLM Inference Acceleration Autoregressive Decoding

Open paper AskPDF arXiv

Speculative decoding accelerates autoregressive generation by using a lightweight draft model to propose multiple tokens that are verified by a target model in parallel. However, the standard acceptance rule focuses on exact distribution correction and rejects tokens that remain highly plausible under the target model when the draft model assigns excess probability. This conservative verification limits the number of draft tokens retained after each verification forward pass. We introduce Nucleus Speculative Decoding (NSD), a relaxed verification method that incorporates target-model plausibility into speculative decoding. NSD accepts a draft token if it satisfies the standard acceptance rule or belongs to the target model's nucleus. We theoretically characterize the distributional deviation introduced by our method and show that the single-step error is exactly determined by the draft model's excess probability within the target nucleus. We further derive sequence-level fidelity bounds that quantify how local deviations accumulate over autoregressive decoding. Experiments across multiple target models and proposal mechanisms demonstrate that NSD consistently improves speculative decoding efficiency while maintaining competitive task performance. Our method achieves throughput speedups of up to

5.16\times

over autoregressive decoding and up to

3.15\times

over standard speculative decoding. These improvements coincide with longer accepted lengths, allowing more output tokens to share the cost of each target verification pass. Analysis shows that plausibility-aware verification provides an effective approach for relaxed verification and speculative decoding efficiency. Our code is available at https://github.com/EIT-NLP/Nucleus-Speculative-Decoding.

Momentum

Related topics

Latest papers 38