Autoregressive Decoding

Follow this topic Ask about this topic

Autoregressive decoding studies how models choose and produce each next token or code from the context generated so far, shaping output quality, diversity, and latency. Common approaches include greedy and beam search, stochastic sampling, constrained generation, and speculative decoding, which drafts several tokens or paths for parallel verification; related methods exploit spatial structure or change decoding order for images and speech. Systems work reduces costs through KV-cache compression, sparse attention, and hardware-aware kernels, with evaluation spanning language reasoning and code tasks, image fidelity, speech quality, and end-to-end throughput. A central tension is whether relaxed verification and added parallelism can deliver real speedups without changing useful output behavior, since exact token matching, task accuracy, and free-running quality do not always agree.

Latest papers

56 papers10 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

Dynamic Multi-Step Decoding for Time-Series Foundation Models
Oct 25, 2026Birmjune Kim, Hyungju Lee, Hyun Oh SongTime Series Forecasting Time-Series Foundation Models
Department of Computer Science and Engineering, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University
ProVe: Proposer-Verifier Decoding
Oct 20, 2026Ziqi Jin, Weiwen Xu, Jiaxi Li +3LLM Inference Acceleration Speculative Decoding
Nanyang Technological University. · ByteDance.
WinARDet: Compressed Window Auto-Regression for Long-Horizon Vision-Language Grounding
Oct 14, 2026Chenxu Wang, Hangyi Kuang, Bo-Wen Yin +6Vision-Language Grounding Autoregressive Decoding
Nankai University · Beijing Normal University
Autoregressive Decoding as a Martingale: Characterizing the Stochastic Dynamics of LLM Generation
Oct 12, 2026Xiahui Zhu, Zhenyi Qi, Wanwei Liu +4Autoregressive Decoding Autoregressive Language Modeling
College of Computer, National University of Defense Technology, Changsha, Hunan, China
Personalized LLM Recommendation with Semantic IDs: Autoregressive Beam Search vs. Multi-Token Prediction
Oct 11, 2026Sejoon Oh, Fernando Amat Gil, Evan Smothers +11Large Language Model-Based Recommendation Semantic ID Recommendation
Netflix Personalization Foundations Team
Lossless Inference Acceleration for ASR via EAGLE-Style Speculative Decoding
Oct 9, 2026Sen Zhang, Xianghu Yue, Yiming Chen +3Automatic Speech Recognition Autoregressive Decoding
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China · ASUS Intelligent Cloud Services, Singapore 486030 · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Guangdong 518172, China +1
AutoIndexer: Flexible-Order Decoding by Training Causal Models on Chains of Edits
Oct 8, 2026Shu Ishida, Aliasghar Khani, Tianyu Zhang +3Language Model Decoding Text Generation
Autodesk Research
Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution
Oct 6, 2026Shuhao Li, Fanghua Ye, Wanyu Lin +2Speculative Decoding LLM Inference Acceleration
EIT-NLP Lab, Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University · Independent Researcher
DLoop: Looped Speculative Decoding
Oct 6, 2026Geonmo Gu, Byeongho Heo, HeeJae Jun +4Speculative Decoding LLM Inference Acceleration
NAVER AI Lab · Korea University · NAVER AI Search Platform
RVQ Position Aware Speculative Decoding for On Device Text to Speech
Sep 29, 2026Berkin Durmus, Eduardo Pacheco, Zach Nagengast +1TTS Synthesis Streaming TTS Synthesis
Argmax, Inc., USA
FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backward
Sep 27, 2026Sriman AchantaSoftmax Attention LLM Inference Acceleration
Virginia Commonwealth University
Fast Plans, Faithful Actions: Closing the Planning-Execution Gap in Hierarchical Vision-Language-Action Models
Sep 25, 2026Chuanliang Xie, Boyu Ma, Gen Li +4Vision-Language-Action Models Efficient VLA Model Inference
MARS Lab, Nanyang Technological University
Adapting Tree-Structured Speculative Decoding to DeepSeek-V4 for Efficient Inference
Sep 21, 2026Changxu Liu, Zhaogeng LiSpeculative Decoding LLM Inference Acceleration
Baige AI Team, Baidu Inc. · Fudan University
One Patch, Three Roles: What Is Actually Coupled in Autoregressive Time-Series Forecasting?
Sep 20, 2026Ziang Li, Yue Huang, Guoxu Zhou +4Long-Term Time Series Forecasting Time Series Forecasting
Guangdong University of Technology · Guangdong Polytechnic Normal University · Harbin Institute of Technology
Evaluating Losslessness in Speculative Decoding Under Finite-Precision Inference
Sep 14, 2026Ilya Koziev, Leonid Sinev, Ivan OseledetsSpeculative Decoding LLM Inference Acceleration
Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference
Sep 13, 2026Tian JinAutoregressive Decoding Efficient Language Model Reasoning
Department of Electrical Engineering and Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY
OneLA: Scaling Linear-Attention Decoding to Large Beams in Generative Recommendation
Sep 11, 2026Xiangrui Yang, Cheng Peng, Yunfeng Zhao +9Linear Attention Autoregressive Decoding
Kuaishou Technology China · Peking University China
Line-Coupled Language Model
Sep 7, 2026Shiyuan Li, Shaorong Zhang, Zhaorui Yang +3Multi-Token Prediction Language Model Decoding
University of California, Riverside Riverside, CA, USA
Masked Autoregressive Speech Enhancement with Continuous Neural Audio Codec Representations
Sep 3, 2026Yoto Fujita, Simon Leglaive, Laurent GirinSpeech Enhancement Autoregressive Decoding
CentraleSup´elec, IETR (UMR CNRS 6164), France · Univ. Grenoble Alpes, CNRS, Grenoble-INP, GIPSA-lab, France
GeoPAR: Large-Scale Multi-Agent Combinatorial Optimization with Geometry-Guided Parallel Autoregressive Learning
Sep 1, 2026Wenjian Wu, Zesheng Jia, Jiaying Tang +2Combinatorial Optimization Vehicle Routing
School of Future Science and Engineering, Soochow University, Suzhou, China
When Should a Network Emit Geometry, and When Should It Detect It? Readout, Reconciliation, and Representation in Floorplan Vectorization
Aug 26, 2026He ZhangIndoor Scene Reconstruction Autoregressive Decoding
Independent Researcher
SHSP: Structure-Aware Hierarchical Solution Prediction for Mixed-Integer Linear Programming
Aug 26, 2026Zherong Zhang, Guanlin Li, Chengrui Gao +5Mixed-Integer Linear Programming Neural Combinatorial Optimization
State Key Laboratory of Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · State Key Laboratory of Technology and Equipment for Defense Against Power System Operational Risks, Nari Technology Co., Ltd.
Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode
Aug 24, 2026Tom PoperszkyEfficient Transformer Inference LLM Inference Acceleration
Independent Research Report
Grounded Decoding for Autoregressive Speech Enhancement via Adaptive Code-Space Grounding and Local LLM Refinement
Aug 21, 2026Hao Shi, Yuan Gao, Zhaoheng Ni +4Speech Enhancement Speech Language Models
Kyoto University · Meta · Brno University of Technology +3
ReLMCodec: Designing Predictable Speech Tokens from Pre-Quantization Phoneme Structure
Aug 8, 2026Zixiang Wan, Xusheng Yang, Zheng Wang +1Neural Audio Codecs Audio Tokenization
1Peking University, Shenzhen, China · 2Tencent, Shenzhen, China
Approximate Speculative Decoding
Aug 4, 2026Yuannuo Feng, Zegang Peng, Yuxin Xie +5Speculative Decoding LLM Inference Acceleration
School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Precision Instrument, Tsinghua University, Beijing, China · Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China +2
From Chains to Trees: Parent-Conditioned Drafting for Semi-Autoregressive Speculative Decoding
Aug 3, 2026Zixian Li, Tong Li, Chi Xie +2Speculative Decoding LLM Inference Acceleration
OPPO AI Center
Wavefront Parallelization for Efficient Learned Image Compression
Jul 21, 2026Shimon Murai, Fangzheng Lin, Kasidis Arunruangsirilert +1Learned Image Compression Inference-Time Optimization
School of Fundamental Science and Engineering Waseda University Tokyo, Japan · School of Engineering Institute of Science Tokyo Tokyo, Japan
Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
Jul 12, 2026Zipeng Gao, Zhi Zheng, Qingrong Xia +5Speculative Decoding LLM Inference Acceleration
1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Huawei Technologies Co., Ltd.
Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inference
Jul 10, 2026Junfei Zhan, Haoxun Shen, Mingang Guo +2Efficient VLM Inference Edge Inference
Department of Electrical and Systems Engineering University of Pennsylvania Philadelphia, PA, USA · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences Shenzhen, China · College of Information Science and Technology Jinan University Guangzhou, China

Oct 25, 2026J/K move · Enter open · S save

Dynamic Multi-Step Decoding for Time-Series Foundation Models

Birmjune Kim, Hyungju Lee, Hyun Oh Song

Department of Computer Science and Engineering, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University

Time Series Forecasting Time-Series Foundation Models Autoregressive Decoding

Open paper AskPDF arXiv

Time-series forecasting models typically use a fixed decoding strategy at inference time: either one-shot prediction of the full horizon or autoregressive rollout over atomic chunks. While one-shot decoding reduces error accumulation, it can discard useful sequential structure; conversely, autoregressive decoding preserves temporal dependencies but may compound errors. We argue that the optimal decoding strategy is not fixed across inputs, but should be selected adaptively for each forecasting instance. We propose a dynamic strategy selector that treats the forecast rollout schedule as an inference-time decision problem for frozen pretrained time-series foundation models. The selector is a learned network that ranks candidate rollout schedules using hidden states of the backbone, and is trained on historical windows of the evaluated tasks for each backbone. To make the candidate set tractable over the exponentially large space of horizon partitions, we use dynamic programming that reduces candidate evaluation to quadratic complexity while preserving much of the oracle gain. At deployment, five independently trained selectors vote on each decision, and the policy follows their consensus, falling back to the backbone's one-shot forecast where they disagree. The construction requires only the backbone's hidden states and lightweight selectors, with no fine-tuning of the forecasting backbone. On fev-bench with a frozen Chronos-2 backbone, agreement-gated selection improves point-forecast skill (MASE) over one-shot decoding with statistical significance, mostly on longer horizons. It recovers about 6-8% of the gain that the same dynamic program attains with ground-truth selection. With a frozen Moirai-1.1-R-small backbone, the improvement is also significant on the full benchmark, and for both backbones we claim gains in point accuracy only.