Language Model Decoding

Ask about this topic

Overview

Language model decoding is the process that turns a model’s next-token predictions into text, and its choices affect accuracy, factuality, style, and speed. Common approaches include greedy, temperature, top-k and nucleus sampling, beam search, and best-of-many selection; newer methods use uncertainty, reward models, verifiers, or retrieved context to guide choices during generation. Work also explores parallel and diffusion-based decoding, speculative generation, and attention or cache optimizations to reduce latency and memory use. A central challenge is balancing quality and reliability against cost and diversity, since confidence and reward signals do not always identify the best continuation and gains can vary by task and evaluation setting.

Latest papers

CardsList

168 papers26 papers in the last four weeks, up 44% on the four weeks before. 0.2% of all new papers.

  1. Beyond End-to-End Recovery: Towards a Better Understanding of LLM Steganography via Rolewise Decomposition

    Oct 21, 2026Om Tailor, Yucheng Pan, Solaleh Mohammadi +2Language Model SteganographyLLM Security

    University of Maryland, College Park

  2. Accept or Repair: Preserving Passage-Supported Answers in Contrastive Decoding

    Oct 21, 2026Jungseob Lee, Seungyoon Lee, Hyeonseok Moon +6Evidence-Grounded QALanguage Model Decoding

    Korea University · Sookmyung Women’s University · Yonsei University Mirae Campus +2

  3. LaVGuard: Selective Decoding for Contextual Value Drift

    Oct 21, 2026Jiale Dai, Liuxian Ma, Hongcan Deng +5LLM AlignmentLanguage Model Decoding

    State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · College of Artificial Intelligence, Tsinghua University · University of Chinese Academy of Sciences +1

  4. When Copying Is Hard: Copy-Constrained Decoding for Exact Span Reproduction

    Oct 20, 2026Jinghui Zhang, Lang Gao, Zongfang Liu +5Language Model DecodingConstrained Decoding

    MBZUAI · Zhejiang University · Westlake University +4

  5. Answering While You Type: Optimizing Time-to-Answer with Streaming Generation

    Oct 19, 2026Nikita Dragunov, Matvey Mikhalchuk, Anton RazzhigaevStreaming InferenceLanguage Model Decoding

    FusionBrain Lab

  6. Keep the Lineages Alive: Diversity-Preserving SMC for Diffusion LLM Decoding

    Oct 19, 2026Minoo Ahmadi, Seyedarmin Azizi, Mehdi Kamal +1Language Model DecodingDiffusion Language Model Decoding

    University of Southern California, Los Angeles, USA

  7. CALM: Consistency-Aware Logit Masking in Decoding Space for Mitigating Overthinking of Large Reasoning Models

    Oct 19, 2026Shengjing Liu, Feiyang Li, Yuxuan Yang +2Overthinking in Language ModelsEfficient Language Model Reasoning

    College of Computer Science and Software Engineering, Shenzhen University

  8. Intent-Triggered Anchoring for Multi-Token Format Units in Parallel Decoding of Diffusion Language Models

    Oct 12, 2026Junhyeok Lee, Kyu Sung ChoiParallel Decoding for Diffusion Language ModelsLanguage Model Decoding

    Interdisciplinary Program in Cancer Biology, Seoul National University College of Medicine · Department of Radiology, Seoul National University Hospital · Department of Radiology, Seoul National University College of Medicine +1

  9. From Candidate Tokens to Final Outcomes: Decoding Relative Long-Horizon Candidate Value from Reasoning Model Representations

    Oct 11, 2026Yixin Yang, Yue Wang, Fandong Meng +1LLM ReasoningLarge Reasoning Models

    State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · WeChat AI, Tencent Inc., China

  10. The Optimal Decoder is Tilted Temperature

    Oct 10, 2026Nikolaos Papagiannis, Ananth Grama, Wojciech SzpankowskiLanguage Model Decoding

    Purdue University

  11. Information-Theoretic Foundations of Online Temperature Sampling

    Oct 10, 2026Jin Sima, Nikolaos Papagiannis, Ananth Grama +1Repetition in Language GenerationLanguage Model Decoding

    Purdue University

  12. Single-Sample Decoding of a Two-Layer Sound-Word Canvas Model for Thai Klon Pad

    Oct 10, 2026Worasait SuwannikText GenerationControllable Text Generation

  13. AutoIndexer: Flexible-Order Decoding by Training Causal Models on Chains of Edits

    Oct 8, 2026Shu Ishida, Aliasghar Khani, Tianyu Zhang +3Language Model DecodingText Generation

    Autodesk Research

  14. SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

    Oct 8, 2026Qitong Wang, Xinwei Niu, Mingluo Su +3LLM PruningLLM Inference Acceleration

    Westlake University · Ant Group

  15. SharedKV-BT: Node-Local Typed Decisions for Behavior-Tree Agents

    Oct 5, 2026Naoki Wake, Justin WagleLanguage Model DecodingRobot Task Planning

    Microsoft

  16. Can Language Models Learn to Reject Their Own Bad Reasoning Steps?

    Oct 5, 2026Siheng Xiong, Xiaoze Liu, Yiqiao Jin +2LLM Self-CorrectionLanguage Model Decoding

    Georgia Institute of Technology · Purdue University

  17. Nash Equilibrium Text: A Game-Theoretic Decoding Framework for Text Generation

    Oct 5, 2026Alireza Jafari, Arman Adibi, Mohammad Ghavamzadeh +1Nash EquilibriumNon-Autoregressive Text Generation

    Department of Computer Science University of Virginia Charlottesville, VA, USA · School of Computer and Cyber Sciences Augusta University Augusta, GA, USA · Qualcomm AI Research San Diego, CA, USA

  18. MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decoding

    Oct 3, 2026Dich Nhat Minh Nguyen, Tran Dang Duong NguyenLanguage Model DecodingLLM Inference Acceleration

  19. Metropolis-Hastings Dominates Importance Resampling for Policy Composition

    Oct 2, 2026Alexey Kurennoy, Ramil Yarullin, Fergal ReidIndependent Metropolis-HastingsLanguage Model Decoding

  20. Decoding Looped Transformers Better for (Almost) Free

    Oct 1, 2026Weihao Liu, Huangjie Zheng, Tianrong Chen +5Efficient Transformer InferenceLanguage Model Decoding

    Apple

  21. Gacha Decoding: Eliciting Diverse Generations Through Instruction Following

    Oct 1, 2026Scott Geng, Yufei Zhang, Joseph Lee +3Language Model DecodingDiverse Text Generation

    University of Washington · Meta Superintelligence Labs

  22. AS2^2D: Accelerating On-Demand Audio Understanding on Mobile Devices

    Sep 29, 2026Yunzhe Li, Kyoungjun Park, Hongzi Zhu +1On-Device Language Model InferenceLanguage Model Decoding

    The University of Texas at Austin · Shanghai Jiao Tong University

  23. Rethinking Soft Tokens for Parallel Decoding in Diffusion Language Models

    Sep 29, 2026Kodai Kawamura, Kenji Kawaguchi, Anji LiuLanguage Model DecodingDiffusion Language Model Decoding

    National University of Singapore

  24. FastGuide: Accelerating Reward Guidance for Diffusion Large Language Models

    Sep 28, 2026Darshan Thaker, Lachlan Ewen MacDonald, René VidalDiffusion Model GuidanceLanguage Model Decoding

    University of Pennsylvania

  25. Twist, Don't Tilt: Trajectory-Exact Constrained Decoding for Masked Diffusion Models

    Sep 28, 2026Aditya Thimmaiah, Lara Marinov, Jayanth Srinivasa +3Constrained DecodingMasked Diffusion Models

    The University of Texas at Austin · Cisco Research

  26. Composable Decoding on the Probability Simplex: Theory and Implementation

    Sep 28, 2026Xiaotong Ji, Ahmed Khaled Khamis, Rasul Tutunov +2Constrained DecodingLanguage Model Decoding

    Huawei Noah’s Ark Lab · UCL Centre for AI

  27. Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding

    Sep 27, 2026Jungseob Lee, Seungyoon Lee, Seongtae Hong +2Language Model DecodingActivation Sparsity

    Korea University · Yonsei University Mirae Campus

  28. Improving the Diversity of LLM Outputs without a Trade-off

    Sep 27, 2026Ryoma SatoOutput Diversity in Language ModelsLanguage Model Decoding

    National Institute of Informatics

  29. Encoded but Not Decoded: Layer-Localized Evidence for a Three-Level Gap in LLM Syntax

    Sep 24, 2026Zhenyan Lu, He Wang, Xiaohui HuangLLM EvaluationLanguage Model Decoding

    College of International Studies, National University of Defense Technology, Nanjing, China

  30. Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs

    Sep 24, 2026Pavel Tikhonov, Anton Korznikov, Matvey Mikhalchuk +6Language Model DecodingDecoder-Only Language Models