Language Model Decoding

Follow this topic Ask about this topic

Language model decoding is the process that turns a model’s next-token predictions into text, and its choices affect accuracy, factuality, style, and speed. Common approaches include greedy, temperature, top-k and nucleus sampling, beam search, and best-of-many selection; newer methods use uncertainty, reward models, verifiers, or retrieved context to guide choices during generation. Work also explores parallel and diffusion-based decoding, speculative generation, and attention or cache optimizations to reduce latency and memory use. A central challenge is balancing quality and reliability against cost and diversity, since confidence and reward signals do not always identify the best continuation and gains can vary by task and evaluation setting.

Latest papers

168 papers26 papers in the last four weeks, up 44% on the four weeks before. 0.2% of all new papers.

Improving reasoning at inference time via uncertainty minimisation
Mar 7, 2026Nicolas Legrand, Kenneth Enevoldsen, Márton Kardos +1Efficient Language Model Reasoning Inference-Time Scaling
Center for Humanities Computing Aarhus University, Denmark
Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decoding
Mar 6, 2026Hoseong Ahn, Jeongyun Chae, Yoonji Park +1Language Model Decoding Automatic Speech Recognition
Sungkyunkwan University, Republic of Korea
Enhancing Instruction Following of LLMs via Activation Steering with Dynamic Rejection
Mar 6, 2026Minjae Kang, Jaehyung KimLanguage Model Steering Instruction Following
Yonsei University
Ensembling Language Models with Sequential Monte Carlo
Mar 5, 2026Robin Shing Moon Chan, Tianyu Liu, Samuel Kiegeland +5Language Model Ensembles Language Model Decoding
ETH Zürich · CHI-FRO · McGill University +2
Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches
Mar 3, 2026Anum Afzal, Yuki Saito, Hiroya Takamura +5Language Model Decoding Multimodal Large Language Models
School of CIT, Technical University of Munich · National Institute of Advanced Industrial Science and Technology (AIST) · The University of Tokyo +3
Enhancing Persona Following at Decoding Time via Dynamic Importance Estimation for Role-Playing Agents
Mar 2, 2026Yuxin Liu, Mingye Zhu, Siyuan Liu +2Large Language Model-Based Role-Play Simulation Persona-Conditioned Generation
University of Science and Technology of China
DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Models
Feb 25, 2026Xi Ye, Wuwei Zhang, Fangcong Yin +2Long-Context Reasoning in Language Models Language Model Decoding
Princeton Language and Intelligence, Princeton University · Department of Computer Science, New York University
Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplers
Feb 20, 2026Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer +1Language Model Decoding Best-of-N Sampling
Huawei Noah’s Ark · AI Centre, Department of Computer Science, UCL
Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model
Feb 6, 2026Jacqueline He, Jonathan Hayase, Wen-tau Yih +3Constrained Decoding Language Model Decoding
University of Washington · Allen Institute for Artificial Intelligence
Attention-Mass Condensation for Sparse Decoding
Feb 6, 2026Jorge L. Ruiz WilliamsLanguage Model Decoding Sparse Attention
NaNZeta LLC
LEAD: Layer-wise Expert-aligned Decoding for Faithful Radiology Report Generation
Feb 4, 2026Ruixiao Yang, Yuanhe Tian, Di Dong +3Radiology Report Generation Language Model Decoding
Beijing Institute of Technology · Zhongguancun Academy · Zhongguancun Institute of Artificial Intelligence +1
Strategies for Span Labeling with Large Language Models
Jan 23, 2026Danil Semin, Ondřej Dušek, Zdeněk KasnerConstrained Decoding Language Model Decoding
Institute of Formal and Applied Linguistics Faculty of Mathematics and Physics, Charles University
LOGIC: Efficient and Robust Contextual Biasing for Speech LLMs via Logit-Space Integration
Jan 21, 2026Peidong Wang, Jian Xue, Jinyu LiLanguage Model Decoding Speech Processing
Microsoft, USA
AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMs
Jan 9, 2026Chengming Cui, Tianxin Wei, Ziyi Chen +6Language Model Decoding Test-Time Scaling
University of Illinois Urbana-Champaign
The Reward Model Selection Crisis in Personalized Alignment
Dec 28, 2025Fady Rezk, Yuangang Pan, Chuan-Sheng Foo +4Personalized Language Model Alignment Personalized Reward Modeling
School of Informatics, University of Edinburgh, UK · Agency for Science, Technology and Research (A*STAR), Singapore · Samsung AI Research Center, Cambridge, UK
ScalingAR: Scaling Confidence for Autoregressive Image Generation
Sep 30, 2025Harold Haodong Chen, Xianfeng Wu, Wen-Jie Shu +4Language Model Decoding Visual Autoregressive Models
HKUST(GZ) · UNC-Chapel Hill · ZODA +3
Evolutionary Guided Decoding: Iterative Value Refinement for LLMs
Mar 4, 2025Zhenhua Liu, Lijun Li, Ruizhe Chen +5LLM Alignment Language Model Decoding
Shanghai Artificial Intelligence Laboratory · Soochow University · Zhejiang University +1
DYNASHIELD: A Black-Box Moving Target Defense for LLMs via Dynamic Decoding Customization
Dec 10, 2024Xiaoqun Liu, Weiming Qi, Qiben YanAdversarial Robustness of Language Models LLM Jailbreak Attacks
Michigan State University, East Lansing, MI 48824, USA

Mar 7, 2026J/K move · Enter open · S save

Improving reasoning at inference time via uncertainty minimisation

Nicolas Legrand, Kenneth Enevoldsen, Márton Kardos, Kristoffer Nielbo

Center for Humanities Computing Aarhus University, Denmark

Efficient Language Model Reasoning Inference-Time Scaling Language Model Decoding

Open paper AskPDF arXiv

Large language models (LLMs) now exhibit strong multi-step reasoning abilities, but existing inference-time scaling methods remain computationally expensive, often relying on extensive sampling or external evaluators. We propose a principled strategy that frames reasoning as uncertainty minimisation and operates at the level of individual thoughts rather than tokens. Our method selects, at each reasoning step, the continuation that maximizes the model's self-certainty, a metric computed from its internal predictive distribution. This approach achieves significant improvement with a small number of samples, relies exclusively on model-internal signals, and applies to open-ended questions as opposed to methods like majority voting. Experiments on MATH500 and GSM8K across multiple model sizes demonstrate that thought-level self-certainty maximization consistently outperforms greedy decoding and matches or exceeds self-consistency under comparable token budgets. Cross-linguistic evaluations further indicate that the method transfers robustly beyond high-resource languages. Furthermore, analysis of self-certainty dynamics reveals that correct reasoning trajectories converge early to stable paths, suggesting that early decisions, likely associated with the planning of the reasoning process, are predictive of final accuracy. Building on this result, we show that self-certainty maximisation applied to the early steps can explain most of the performance gain and provide a simple yet efficient inference-time scaling method.