Language Model Decoding

Ask about this topic

Overview

Language model decoding is the process that turns a model’s next-token predictions into text, and its choices affect accuracy, factuality, style, and speed. Common approaches include greedy, temperature, top-k and nucleus sampling, beam search, and best-of-many selection; newer methods use uncertainty, reward models, verifiers, or retrieved context to guide choices during generation. Work also explores parallel and diffusion-based decoding, speculative generation, and attention or cache optimizations to reduce latency and memory use. A central challenge is balancing quality and reliability against cost and diversity, since confidence and reward signals do not always identify the best continuation and gains can vary by task and evaluation setting.

Latest papers

CardsList

168 papers26 papers in the last four weeks, up 44% on the four weeks before. 0.2% of all new papers.

  1. Efficient Test-Time Inference via Deterministic Exploration of Truncated Decoding Trees

    Apr 22, 2026Xueyan Li, Johannes Zenn, Ekaterina Fadeeva +3Language Model DecodingInference-Time Search

    Max Planck Institute for Intelligent Systems · ETH Zurich · AI Center Tübingen +3

  2. LatticeBridge: Rare-Event Sequential Inference for Faithful Structured Sequence Synthesis

    Apr 22, 2026Faruk Alpay, Bugra KilictasConstrained DecodingGrounded Text Generation

    Department of Computer Engineering, Bahcesehir University

  3. DocQAC: Adaptive Trie-Guided Decoding for Effective In-Document Query Auto-Completion

    Apr 20, 2026Rahul Mehta, Kavin R, Indrajit Pal +3Language Model Decoding

    Microsoft Corporation Hyderabad, India · Indian Institute of Technology Kharagpur, India · Independent Bengaluru, India

  4. No-Worse Context-Aware Decoding: Preventing Neutral Regression in Context-Conditioned Generation

    Apr 17, 2026Yufei Tao, Ameeta AgrawalLanguage Model DecodingLLM Reliability

    Department of Computer Science, Portland State University, USA

  5. Min-kk Sampling: Decoupling Truncation from Temperature Scaling via Relative Logit Dynamics

    Apr 13, 2026Yuanhao Ding, Meimingwei Li, Esteban Garces Arias +3Language Model DecodingAdaptive Inference

    School of Computer and Information Engineering, Henan University · Department of Statistics, LMU Munich · Munich Center for Machine Learning (MCML)

  6. Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation

    Apr 11, 2026Yebo Wu, Han Jin, Zhijiang Guo +1VLM Hallucination MitigationLanguage Model Decoding

    State Key Laboratory of IOTSC, University of Macau · Hong Kong University of Science and Technology (Guangzhou)

  7. Process Reward Agents for Steering Knowledge-Intensive Reasoning

    Apr 10, 2026Jiwoong Sohn, Tomasz Sternal, Kenneth Styppa +2LLM ReasoningProcess Reward Models

    ETH Zürich, Switzerland · Heidelberg University, Germany

  8. GCoT-Decoding: Unlocking Deep Reasoning Paths for Universal Question Answering

    Apr 8, 2026Guanran Luo, Wentao Qiu, Zhongquan Jian +2LLM ReasoningLanguage Model Decoding

    School of Informatics, Xiamen University

  9. Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo

    Apr 7, 2026Jelena Markovic-Voronov, Wenhui Zhu, Bo Long +5Language Model DecodingLLM Code Generation

    LinkedIn

  10. Embarrassingly Simple Self-Distillation Improves Code Generation

    Apr 1, 2026Ruixiang Zhang, Richard He Bai, Huangjie Zheng +3Language Model DecodingLanguage Model Distillation

    Apple

  11. CASA: Classification Augmented with Safety Attention for Robust Multimodal Alignment

    Mar 31, 2026Anurag Kumar, Raghuveer Peri, Jon Burnsky +4MLLM SafetyLLM Safety Alignment

    The Ohio State University · AWS

  12. The Necessity of Setting Temperature in LLM-as-a-Judge

    Mar 30, 2026Lujun Li, Lama Sleem, Yangjie Xu +4LLM-as-a-JudgeLLM Evaluation

    University of Luxembourg · ETH Zürich

  13. ATLAS-RTC: Closing the Loop on LLM Agent Output with Token-Level Runtime Control

    Mar 29, 2026Christopher CruzLanguage Model DecodingStructured Output Generation

    Purdue University

  14. Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs

    Mar 26, 2026Vishal Narnaware, Animesh Gupta, Kevin Zhai +2Language Model DecodingMultimodal Grounding

    Institute of Artificial Intelligence, University of Central Florida

  15. Pingala: Prosody-Aware Decoding for Sanskrit Poetry Generation

    Mar 25, 2026Manoj Balaji Jagadeeshan, Atul Singh, Nallani Chakravartula Sahith +2Text GenerationLanguage Model Decoding

    Indian Institute of Technology, Kharagpur · Oriflow

  16. Inducing Sustained Creativity and Diversity in Large Language Models

    Mar 19, 2026Queenie Luo, Gary King, Michael Puett +1Language Model DecodingDiverse Text Generation

    Ph.D. candidate, Department of East Asian Languages and Civilizations, Harvard University · Albert J. Weatherhead III University Professor, Institute for Quantitative Social Science, Harvard University · Walter C. Klein Professor of Chinese History and Anthropology, Harvard University +1

  17. D5P4: Partition Determinantal Point Process for Diversity in Parallel Discrete Diffusion Decoding

    Mar 19, 2026Jonathan Lys, Vincent Gripon, Axel Marmoret +4Diffusion Language Model DecodingLanguage Model Decoding

    IMT Atlantique, Lab-STICC, UMR CNRS 6285, F-29238 Brest, France. · Sony Europe Ltd. Stuttgart Technology Center, EUREC, Germany.

  18. The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choices

    Mar 19, 2026Esteban Garces Arias, Nurzhan Sapargali, Christian Heumann +1AI-Generated Text DetectionLanguage Model Decoding

    Department of Statistics, LMU Munich · Munich Center for Machine Learning (MCML)

  19. Adaptive Decoding via Test-Time Policy Learning for Self-Improving Generation

    Mar 19, 2026Asmita Bhardwaj, Yuya Jeremy Ong, Eelaaf Zahid +1Language Model DecodingTest-Time Adaptation

    UC San Diego San Diego, CA · Plastic Labs San Jose, CA · IBM Research Cambridge, MA +1

  20. Temperature-Dependent Performance of Prompting Strategies in Extended Reasoning Large Language Models

    Mar 18, 2026Mousa Salah, Amgad MuneerLLM PromptingLLM Reasoning

    Department of Computer Engineering, Gujarat Technological University, India · The University of Texas MD Anderson Cancer Center, Texas, USA

  21. FlashSampling: Fast and Memory-Efficient Exact Sampling

    Mar 16, 2026Tomas Ruiz, Zhen Qin, Yifan Zhang +3LLM Inference AccelerationGPU Kernel Optimization

    LMU Munich · FlashSampling · Princeton University

  22. DOS: Dependency-Oriented Sampler for Masked Diffusion Language Models

    Mar 16, 2026Xueyu Zhou, Yangrong Hu, Jian HuangLanguage Model DecodingDiffusion Language Model Decoding

    Department of Data Science and AI The Hong Kong Polytechnic University, Hong Kong, China

  23. LightBeam: An Accurate and Memory-Efficient CTC Decoder for Speech Neuroprostheses

    Mar 14, 2026Ebrahim Feghhi, Junlin Hu, Nima Hadidi +1Connectionist Temporal ClassificationBrain-Computer Interfaces

    Neuroscience Interdepartmental Program, University of California Los Angeles · Department of Electrical & Computer Engineering, University of California Los Angeles · Department of Computer Science, University of California Los Angeles

  24. Towards unified brain-to-text decoding across speech production and perception

    Mar 13, 2026Zhizhang Yuan, Yang Yang, Gaorui Zhang +7Brain-to-Speech DecodingNeural Decoding

    Computer Science and Technology, Zhejiang University, Hangzhou, Zhejiang, China. · Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, China.

  25. Compositional Consistency-Guided Decoding for Three-Way Logical Question Answering

    Mar 12, 2026Tianyi Huang, Ming Hou, Jiaheng Su +2Question AnsweringLogical Reasoning

    Ryquo · App-In Club

  26. BLooP: Zero-Shot Abstractive Summarization using Large Language Models with Bigram Lookahead Promotion

    Mar 12, 2026Varun Iyer, Cornelia CarageaText SummarizationLanguage Model Decoding

    University of Illinois Chicago

  27. Think Twice Before You Write -- an Entropy-based Decoding Strategy to Enhance LLM Reasoning

    Mar 10, 2026Jiashu He, Meizhu Liu, Olaitan P Olaleye +9Language Model DecodingEfficient Language Model Reasoning

    Oracle AI Science · University of Pennsylvania

  28. Learning Adaptive LLM Decoding

    Mar 10, 2026Chloe H. Su, Zhe Ye, Samuel Tenka +3Language Model DecodingInference-Time Adaptation of Language Models

    Department of Computer Science, Harvard University, Boston, USA · Kempner Institute, Harvard University, Cambridge, MA, USA · UC Berkeley, CA, USA +1

  29. How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms

    Mar 9, 2026JV RoigHallucination in Language ModelsLLM Evaluation

    Kamiwaza AI

  30. DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding

    Mar 9, 2026Mingxi Zou, Jiaxiang Chen, Junfan Li +4Preference AlignmentInference-Time Language Model Alignment

    Fudan University, Shanghai, China · Independent Researcher · Meta AI +1