Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Models

Haoran Wang; Xiongxiao Xu; Philip S. Yu; Kai Shu

cs.CLAug 14, 2026

Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Models

Authors: Haoran Wang, Xiongxiao Xu, Philip S. Yu, Kai Shu

Organizations: Emory University · Illinois Institute of Technology · University of Illinois Chicago

Abstract

Large language models (LLMs) and large vision-language models (LVLMs) have demonstrated impressive generative capabilities, yet ensuring their outputs align with user intent is still challenging. While most existing approaches address this issue at the training stage, inference-time approaches like decoding methods offer a more efficient and scalable solution. Decoding methods control model generation by guiding token-level selection, performing sequence-level generation, or generating tokens in parallel to accelerate the process. In this survey, we identify three emerging paradigms from recent works on decoding methods for LLMs and LVLMs, provide a systematic review of these methods, highlight ongoing challenges, and discuss potential future research directions. Our goal is to underscore the efficiency and effectiveness of decoding methods and offer a practical view of their applications. Paper lists and more resources on decoding methods for LLMs and LVLMs can be found at https://github.com/wang2226/Awesome-LLM-Decoding.

Figures & tables

Figure 2: Timeline illustrating the evolution of decoding methods for LLMs and LVLMs.

Figure 3: Illustration of different decoding paradigms. Contrastive decoding selects the next token by maximizing the contrast between two underlying probability distributions. Guided decoding determines the next token based on the highest score from a guidance function. Parallel decoding generates multiple token candidates simultaneously and selects the most probable one.

Paradigm	Work	Description	Model	Year ( $\downarrow$ )
Contrastive Decoding	DExpert [ 73 ]	Detoxification with contrastive decoding	PLM	2021
	CD [ 65 ]	Formulate Contrastive Decoding	PLM	2022
	CAD [ 116 ]	Context-aware Decoding	LLM	2023
	SCD [ 166 ]	Speculative contrastive decoding	LLM	2023
	DoLa [ 20 ]	Decoding by contrasting layers	LLM	2023
	VCD [ 59 ]	Visual contrastive decoding	LVLM	2024

Table 1: List of works categorized by the three paradigms identified in this survey. Based on chronological order.

Language Model Decoding

Language model decoding studies how models choose and generate outputs, balancing quality, reliability, diversity, and inference cost.

See all 168 papers

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
Oct 8, 2026Qitong Wang, Xinwei Niu, Mingluo Su +3LLM Pruning LLM Inference Acceleration
Westlake University · Ant Group
Beyond Truncation: Rethinking LLM Decoding as Ensemble Pruning
Sep 16, 2026Dunyao Xue, Chengshuo Du, Zhengbo Wang +2Language Model Decoding Diverse Text Generation
Institute of Statistics and Big Data, Renmin University of China, Beijing, China · Big Data and Responsible Artificial Intelligence for National Governance, Renmin University of China, Beijing, China · Center for Applied Statistics, Institute of Statistics and Big Data, Renmin University of China, Beijing, China
Learning Adaptive LLM Decoding
Mar 10, 2026Chloe H. Su, Zhe Ye, Samuel Tenka +3Language Model Decoding Inference-Time Adaptation of Language Models
Department of Computer Science, Harvard University, Boston, USA · Kempner Institute, Harvard University, Cambridge, MA, USA · UC Berkeley, CA, USA +1

Oct 8, 2026J/K move · Enter open · S save

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

Qitong Wang, Xinwei Niu, Mingluo Su, Shanwei Zhao+2

Westlake University · Ant Group

LLM Pruning LLM Inference Acceleration Language Model Decoding Structured Sparsity

Open paper AskPDF arXiv

The memory-bound nature of the decoding stage of large language model (LLM) inference incurs significant latency. Layer-wise training-free network pruning approaches guided by the Hessian have been a prominent solution to this problem, as pruning reduces the number of nonzero parameters read from memory during decoding. Nevertheless, typical methods in this line compute the Hessian using pre-collected natural sequences, whereas the model is fed self-generated tokens during decoding, creating a distribution shift between the two sequences. The Hessian calculated on the natural sequence is different from that calculated on the generated sequence. We observe that this discrepancy causes the activation distribution during generation to deviate from that used for pruning, further hurting the pruned model performance. Moreover, most existing LLM pruning methods that bring actual speedup primarily target the sparse matrix-matrix (SpMM) multiplication, providing limited support for the sparse matrix-vector (SpMV) operations, which dominate decoding. To solve these problems, we introduce SparseDecoding, a principled decoding-aware pruning framework tailored for accurate and efficient LLM decoding. Specifically, at the algorithmic axis, SparseDecoding constructs calibration matrices from layer-wise activations collected during the dense-model autoregressive generation, excluding prefill, thereby aligning the pruning objective with the decoding activations. At the system axis, we develop an optimized N:M sparse matrix-vector kernel with bitmask indexing and fixed-step traversal. Substantial empirical results on representative LLMs (Llama-3.1-8B, Llama-3.3-70B, Qwen3-14B / 32B) demonstrate that our method consistently outperforms standard fixed-text calibration on the long-form generation benchmarks while achieving up to 1.48x end-to-end wall-clock decoding speedup on A100 GPUs.

Abstract

Language Model Decoding

Explore similar work

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

Beyond Truncation: Rethinking LLM Decoding as Ensemble Pruning

Learning Adaptive LLM Decoding