Next Token or Vector? Learning How Far to Decode

Sai Sreenivas Chintha; Chidera Biringa; Rishab Sharma; Anusha Agrawal; Prachi Badarayani; Gemma Zhang; Nigel Boachie Kumankumah; Marco Ayala

cs.LGOct 16, 2026

Next Token or Vector? Learning How Far to Decode

Authors: Sai Sreenivas Chintha, Chidera Biringa, Rishab Sharma, Anusha Agrawal, Prachi Badarayani, Gemma Zhang, Nigel Boachie Kumankumah, Marco Ayala,

Organizations: Microsoft · University of Massachusetts Amherst

Abstract

Standard autoregressive decoding advances one token at a time, even when several upcoming tokens can be predicted together. We propose CADENCE (Context-Adaptive DEcoding), a generation method that learns both what to draft and how far to draft. Gaussian prediction heads represent future spans as continuous latent vectors, and a shared decoder converts the selected vector into token predictions in parallel. A learned router selects the prediction length from the model's own hidden states and predicted Gaussian parameters, adapting generation granularity to the current context with single-token generation as a fallback. The backbone verifies each proposed continuation and retains the prefix that matches its greedy predictions. Experiments with Llama-3.1-8B-Instruct on Alpaca, GSM8K, and HumanEval show that router-selected positions yield higher draft acceptance than applying the corresponding prediction length everywhere. CADENCE achieves up to 1.3x higher generation throughput than standard greedy decoding.

Figures & tables

Figure 2: Overview of the CADENCE architecture and training framework. The backbone provides context representations for Gaussian heads that predict future-span latents. A shared decoder generates token predictions, while a router selects the prediction horizon using the context and Gaussian parameters. Training uses Gaussian negative log-likelihood (NLL), KL self-distillation, and full-span matching supervision for the router.

Method	Prediction	Length control
CALM	Chunk vector	Fixed chunk size
CoLaR	Reasoning latent	Prompted compression
MEDUSA	Token tree	Configured tree
EAGLE-3	Token tree	Confidence-based tree selection
DFlash	Token block	Configured block size
SpecDec++	Token chain	Learned stopping

Table 1: Prediction forms and length-control mechanisms.

Figure 3: Token acceptance rates by prediction horizon under greedy verification on GSM8K, HumanEval, and Alpaca. Solid lines report positions where the router selected each horizon; dashed lines report the corresponding horizon applied at every evaluated position.

Figure 5: Share of generation steps by the number of tokens retained after greedy verification for the router. One-token steps include both base-model fallbacks and verified drafts for which only one token is retained.

		Context-only router			Gaussian-enhanced router
Horizon	Draft	$\tau$	Token acc.	Full acc.	$\tau$	Token acc.	Full acc.
$k=1$	2	1.70	84.8%	69.5%	1.74	87.1%	74.3%
$k=2$	3	2.27	75.5%	49.8%	2.38	79.3%	58.6%
$k=3$	4	2.68	67.1%	35.4%	2.74	68.5%	40.0%
$k=4$	5	2.85	56.9%	21.5%	3.02	60.5%	31.3%
$k=5$	6	3.27	54.6%	25.1%	3.57	59.4%	30.7%

Table 2: Effect of Gaussian prediction parameters on adaptive routing on Alpaca under greedy verification. Per-horizon metrics are computed over router-selected drafts; overall metrics include backbone-only fallback steps. Bold indicates the better result.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	Exact-match label
Router	0	1	2	3	4	5
0	35.0	12.3	4.4	1.1	0.3	0.1
1	3.9	13.8	3.3	1.0	0.3	0.1
2	2.4	3.2	6.8	1.5	0.4	0.1
3	1.1	1.1	1.0	2.1	0.4	0.1
4	0.2	0.4	0.2	0.2	0.6	0.1

Appendix

Table 4: Router prediction against the exact-match label for the UltraChat router, at every token of 80 Alpaca greedy answers (17,359 positions). Each cell is the percentage of positions with that pair. Class 0 is no full match. Bold cells are exact agreement, and darker blue is a larger share of positions.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,033 papers

RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding
Apr 16, 2026Zihong Zhang, Zuchao Li, Lefei Zhang +2Speculative Decoding LLM Inference Acceleration
School of Artiﬁcial Intelligence, Wuhan University, Wuhan, China · School of Computer Science, Wuhan University, Wuhan, China · School of Information Management, Wuhan University, Wuhan, China +1
Entropy-informed Decoding: Adaptive Information-Driven Branching
May 10, 2026Benjamin Patrick Evans, Sumitra Ganesh, Leo ArdonLanguage Model Decoding LLM Inference Acceleration
JP Morgan AI Research, London, UK · JP Morgan AI Research, New York, USA
DLoop: Looped Speculative Decoding
Oct 6, 2026Geonmo Gu, Byeongho Heo, HeeJae Jun +4Speculative Decoding LLM Inference Acceleration
NAVER AI Lab · Korea University · NAVER AI Search Platform

Apr 16, 2026J/K move · Enter open · S save

RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding

Zihong Zhang, Zuchao Li, Lefei Zhang, Ping Wang+1

School of Artiﬁcial Intelligence, Wuhan University, Wuhan, China · School of Computer Science, Wuhan University, Wuhan, China · School of Information Management, Wuhan University, Wuhan, China+1

Speculative Decoding LLM Inference Acceleration Autoregressive Decoding

Open paper AskPDF arXiv

Autoregressive decoding in Large Language Models (LLMs) generates one token per step, causing high inference latency. Speculative decoding (SD) mitigates this through a guess-and-verify strategy, but existing training-free variants face trade-offs: retrieval-based drafts break when no exact match exists, while logits-based drafts lack structural guidance. We propose

\textbf{RACER}

(

\textbf{R}

etrieval-

\textbf{A}

ugmented

\textbf{C}

ont

\textbf{e}

xtual

\textbf{R}

apid Speculative Decoding), a lightweight and training-free method that integrates retrieved exact patterns with logit-driven future cues. This unification supplies both reliable anchors and flexible extrapolation, yielding richer speculative drafts. Experiments on Spec-Bench, HumanEval, and MGSM-ZH demonstrate that RACER consistently accelerates inference, achieving more than

2\times

speedup over autoregressive decoding, and outperforms prior training-free methods, offering a scalable, plug-and-play solution for efficient LLM decoding. Our source code is available at https://github.com/hkr04/RACER.

Abstract

LLM Inference Acceleration

Explore similar work

RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding

Entropy-informed Decoding: Adaptive Information-Driven Branching

DLoop: Looped Speculative Decoding