Recycling a target model's own predictions offers a training-free source of proposals for speculative decoding. In vision-language models, however, existing recycling indexes predictions by the last token alone, conflating predictions computed under different images and prefixes; we find that such reuse fails most often at image-sensitive decoding states. Longer keys reduce this aliasing but match fewer states. We introduce \method, which stores target predictions under one-, two-, and three-token keys: contextual memories live only for the current request and are thus implicitly conditioned on its image, while token-level memory persists across requests to retain coverage. At each lookup, entropy-confidence weighting fuses the retrieved predictions according to their key specificity and concentration. Best-first search composes the fused transitions into a candidate tree, supplemented by suffix-matched continuations, and one target pass verifies the tree under greedy decoding. Without training or an auxiliary draft model, \method\ achieves mean speedups over autoregressive decoding of 1.66\speed and 1.56\speed across seven single-turn benchmarks on Qwen2.5-VL-7B and LLaVA-1.6-7B, exceeding the strongest training-free baseline, Token Recycling (1.51\speed and 1.37\speed). Our implementation is available in https://github.com/Senmo996/RECAP-official-code.
Figures & tables
Figure 1: The role of context in prediction recycling. Top: a cached her → head transition fails when reused after a different prefix. Middle: shorter accepted paths accompany stronger root image sensitivity (left), and source preference varies with candidate visual alignment (right). Bottom: Recap retrieves and fuses predictions at multiple contextual resolutions.
Figure 2: The Recap framework. (A) Three-, two-, and one-token keys retrieve Top- K prediction rows from G , C , and U ; only U survives request boundaries. (B) Entropy-confidence weighting fuses the retrieved rows under specificity priors and normalizes the retained candidates. (C) Best-first tree search repeats retrieval and fusion under each node’s path context. (D) Suffix retrieval inserts a contiguous continuation from the prompt and committed history. (E) One packed target pass commits the matching path plus one target-selected token. All evaluated nodes refresh prediction memory; only committed tokens extend sequence history. Tokens and scores are illustrative.
Models
Methods
MMT
SEED
SciQA
OCR
ChartQA
MathVista
TextVQA
Average
S
τ
S
τ
S
τ
S
τ
S
τ
S
τ
S
τ
S
τ
Qwen2.5-VL 7B-Instruct
PLD
1.16
1.19
0.89
1.05
1.24
1.13
1.22
1.20
1.14
1.25
1.49
1.22
0.97
1.09
1.16
1.16
Lookahead
1.15
1.60
0.93
1.35
1.44
2.14
1.11
1.62
1.10
1.62
1.20
1.60
0.94
1.26
1.12
1.60
Tree Recycling
1.20
1.47
1.07
1.49
1.23
1.54
1.29
1.68
1.27
1.72
1.41
1.70
1.06
1.41
1.22
1.57
Token Recycling
1.59
2.57
1.25
2.58
1.60
2.60
1.51
2.82
1.54
2.89
1.86
2.84
1.21
2.63
1.51
2.71
SAM
1.11
1.24
1.06
1.34
1.15
1.36
1.20
1.33
1.23
1.57
1.21
1.38
1.05
1.26
1.14
1.35
Table 1: Speedup S ( × ) and acceptance length τ across seven single-turn benchmarks at temperature 0 . SciQA denotes ScienceQA and OCR denotes OCRBench. S averages paired generation-throughput ratios against autoregressive decoding; τ averages accepted draft tokens plus one target-selected token. Higher is better for both metrics.
Models
Methods
General VQA
Text VQA
Captioning
Charts
Reasoning
Multi-turn
Overall
S
τ
S
τ
S
τ
S
τ
S
τ
S
τ
S
τ
Qwen2.5-VL 7B-Instruct
PLD
1.06
1.05
1.21
1.13
1.14
1.15
1.20
1.31
1.28
1.34
1.34
1.27
1.23
1.22
Lookahead
1.11
1.37
1.04
1.29
1.44
1.64
1.21
1.77
1.91
2.28
1.39
1.80
1.36
1.71
Tree Recycling
1.12
1.40
1.10
1.32
1.22
1.37
1.21
1.64
1.38
1.66
1.25
1.56
1.22
1.51
Token Recycling
1.51
2.57
1.31
2.49
1.80
2.51
1.38
2.60
1.77
2.64
1.57
2.60
1.56
2.57
SAM
1.05
1.14
1.13
1.13
1.03
1.07
1.14
1.54
1.18
1.31
1.08
1.23
1.10
1.24
Table 2: MMSpec speedup S ( × ) and acceptance length τ at temperature 0 , with the metric definitions of Table 1 . Each category contains 100 inputs; multi-turn conversations contribute 223 generation turns, giving 723 total turns. Overall values are turn-weighted averages of both metrics, rather than unweighted category means.
Model
Method
Params
GiB
τ
S
AR
DREAM
607M
1.13
3.68
1.26
41.9
ViSpec
451M
0.84
3.74
1.05
61.2
LLaVA
Recap
0
0
2.92
1.56
29.9
ViSpec
890M
1.66
3.76
1.18
57.0
Qwen
Recap
0
0
3.02
1.66
39.4
Table 3: Deployment cost and decoding performance on the seven single-turn benchmarks in Table 1 . Params and GiB denote drafter parameter counts and 16-bit GPU weight storage; AR is mean autoregressive throughput (tokens/s). DREAM and ViSpec use released checkpoints with top- 4 , depth- 6 , 32 -node trees and a faster AR harness, so their S is not directly comparable with that of Recap .
Figure 3: The specificity–coverage trade-off and contextual backoff. (a) Query coverage and conditional Recall@ 8 across 23,811 states from 400 inputs. Each benchmark connects U (circle), C (square), and G (diamond). Finer keys raise conditional recall while reducing coverage. (b) Paired online throughput gains of U+C backoff against U .
Comparison
Gain (%, ↑ )
Candidate search
Best-first vs. fixed expansion
+4.90
Memory design
Context backoff vs. unigram
+3.03
Uniform fusion vs. backoff
+0.75
Specificity priors vs. uniform fusion
+2.80
Table 4: Paired throughput effects of individual components. Unmarked rows use the 120 -input MMSpec subset with Qwen2.5-VL-7B; the persistence row averages five runs. † marks four-dataset controls ( 200 inputs per model for confidence, reported as Qwen/LLaVA; 400 inputs for suffix insertion). Backoff selects the most specific row available at a query.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Coverage (%, ↑ )
Recall@ 8 (%, ↑ )
U
C
G
U
C
G
ChartQA
44.5
23.2
14.8
64.5
81.7
86.6
TextVQA
26.3
7.4
3.4
52.5
73.5
78.3
OCRBench
54.9
30.4
18.6
63.6
84.2
90.6
MMT-Bench
45.6
19.7
11.2
56.9
77.9
84.4
Appendix
Table 5: Coverage and conditional Recall@ 8 during empty-initialization replay. Queries are evaluated before their corresponding memory updates.
Policy
Matched continuation length ( ↑ )
ChartQA
TextVQA
OCR
MMT
U
0.516
0.224
0.632
0.444
U+C backoff
0.709
0.253
1.135
0.600
U+C+G backoff
0.778
0.252
1.128
0.638
Uniform fusion
0.737
0.253
1.041
0.629
Prior-weighted fusion
0.751
0.253
1.066
0.639
Appendix
Table 6: Matched continuation length in empty-initialization replay. Each state uses equal actual node counts across the compared policies. Prior-weighted fusion uses fixed priors ( cj=1 ).
Dataset
Offline matched continuation length
Online fusion + suffix / fusion
Fusion
+ Suffix
Difference
Gain (%, ↑ )
τ : before → after
ChartQA
1.123
1.313
0.190
+0.83
2.80→2.88
TextVQA
0.355
0.413
0.059
+0.07
2.18→2.18
OCRBench
1.417
1.538
0.120
−0.67
2.66→2.62
MMT-Bench
0.750
0.866
0.116
+2.23
2.40→2.44
Appendix
Table 7: Suffix-retrieval controls. Offline matching omits root and correction tokens; online gains and acceptance length τ use Equations 9 – 10 .
Figure 4: Token-level reuse across task schedules. Persistent U is paired with per-request reset on identical inputs to measure throughput under blocked and alternating benchmark streams.
Dataset
Throughput gain (%, ↑ )
Blocked
Alternating
ChartQA
5.69±0.59
3.91±0.77
TextVQA
0.92±1.10
0.68±1.13
OCRBench
4.77±1.04
4.32±0.13
MMT-Bench
7.98±1.12
7.49±1.77
Appendix
Table 8: Mean throughput gain ± sample SD for persistent U versus reset, measured across three orders of the same 200 inputs.
Qwen
LLaVA
Stage
Wall
CPU
Wall
CPU
Suffix lookup
0.010
0.016
0.010
0.016
Lookup, fusion, search
0.506
0.510
0.592
0.598
Packing
0.391
0.409
0.519
0.538
Memory refresh
0.461
0.468
0.414
0.421
Suffix index update
0.027
0.034
0.028
0.034
Appendix
Table 9: Decoding-phase times in milliseconds, pooled over iterations at a 63 -node cap with 40 inputs per model and two orders. Wall times are exclusive; CPU time includes synchronization polling.
Model
Cap
Mean nodes
Prep. ms
p95 ms
Host MiB
GPU GiB
Qwen
31
26.5
0.591
0.722
5.79
19.32
Qwen
63
55.2
0.907
1.227
7.88
19.32
Qwen
127
113.4
2.202
2.617
8.74
19.32
LLaVA
31
29.1
0.721
0.882
7.46
19.98
LLaVA
63
59.2
1.121
1.385
8.70
20.01
LLaVA
127
121.0
2.485
2.729
12.68
20.16
Appendix
Table 10: Budget sensitivity with shared inputs and orders. Candidate counts include suffixes and exclude the root. Preparation covers lookup through packing; p95 is the request-average 95th percentile. Host memory is the largest proposal-state snapshot. GPU memory is peak model-process allocation, including weights and generation workspaces.
Model
∣U∣
Host MiB
Lookup ns
Qwen
1,000
0.67
109
10,000
7.03
141
152,064
109.14
386
LLaVA
1,000
0.67
109
10,000
7.03
140
32,064
23.36
183
Appendix
Table 11: Synthetic occupancy stress test: host-object storage and median lookup latency, measured over seven repeats of 100,000 seeded ID/score hits with a checksum.
Variant
S ( × , ↑ )
τ ( ↑ )
Recap (lexical)
1.63
2.67
Recap + semantic fallback
1.61
2.78
Recap + recency fallback
1.61
2.75
Appendix
Table 12: Root fallback on MMSpec with Qwen2.5-VL-7B. All variants use fixed-prior fusion ( cj=1 ). Longer accepted continuations accompany lower speedup. Best column values are bold.
Variant
Qwen
LLaVA
ChartQA
TextVQA
OCR
MMT
Mean
τ
ChartQA
TextVQA
OCR
MMT
Mean
τ
Recap
+0.65
+1.24
+1.43
−0.41
+0.73
2.93
+2.30
+0.34
+1.71
+0.01
+1.09
2.90
w/o confidence
+0.00
+0.00
+0.00
+0.00
+0.00
2.91
+0.00
+0.00
+0.00
+0.00
+0.00
2.88
w/o prior
+0.37
+0.56
+1.38
−0.93
+0.35
2.91
+0.26
−0.67
+0.31
−0.92
−0.25
2.85
Uniform
+0.01
+0.37
+2.34
+0.22
+0.73
2.90
−0.47
+0.18
+2.29
−0.06
+0.48
2.86
Confidence gate
−1.10
+1.52
−1.33
−1.72
−0.66
2.83
−2.27
−1.56
−1.25
−2.53
−1.90
2.76
Appendix
Table 13: Online fusion ablations, reporting mean input-paired throughput changes (%) against w/o confidence and acceptance length τ . Each model processes 200 inputs, 50 per dataset, under two orders. Policies share memory scope, candidate limits, and suffix insertion; timing includes weighting. Full Recap is shaded.
Variant
ChartQA
TextVQA
OCR
MMT
Recap
+0.00
+0.00
+0.01
−0.01
w/o confidence
+0.00
+0.00
+0.00
+0.00
w/o prior
−0.08
+0.00
−0.31
+0.00
Uniform
−0.08
+0.00
−0.30
+0.02
Confidence gate
−1.49
−0.14
−1.99
−0.95
Specificity backoff
−0.11
+0.00
+1.29
−0.24
Appendix
Table 14: Changes in all-query Recall@ 8 relative to w/o confidence (percentage points), evaluated on 23,811 Qwen states from 400 inputs. Uncovered states count as misses. Shading identifies Recap .
Figure 5: Persistent U over cyclic replay of the same 400 trajectories. Curves report the mean and min–max range across three replay orders.
Figure 6: Per-input generation-latency CDFs on MMSpec for paired Recap and AR runs. A leftward curve indicates lower latency. Multi-turn inputs contribute summed generation time across turns.
Model
Method
P50
P90
P95
P99
Qwen
AR
2.70
8.44
11.63
15.07
Qwen
Recap
1.52
4.46
6.08
7.56
LLaVA
AR
4.02
9.42
14.27
18.43
LLaVA
Recap
2.38
5.45
8.25
10.67
Appendix
Table 15: MMSpec per-input generation-latency percentiles in seconds, with lower values indicating faster generation. P k denotes the k th percentile; Recap rows are shaded.
Figure 7: Fusion gains over confidence gating on Qwen replay states with source disagreement. Points show Recall@ 8 differences; bars give 95% confidence intervals from 2,000 paired image-cluster bootstrap resamples. The dashed line marks zero gain, and n is the number of disagreement states.
Figure 8: Per-input throughput speedup on the seven single-turn benchmarks ( 1,000 inputs per model and benchmark). Points mark medians and segments span P10–P90 of the input-level ratios. Dashed lines mark parity with AR; the percentages report inputs with strictly higher throughput than paired AR.
Dataset
States
Recap
Confidence gate
Difference
95% CI
ChartQA
427
84.31
66.98
+17.33
[12.08,22.31]
TextVQA
52
75.00
65.38
+9.62
[−2.33,18.42]
OCRBench
857
62.43
46.91
+15.52
[9.69,20.70]
MMT-Bench
510
77.65
62.55
+15.10
[10.27,20.14]
Appendix
Table 16: Recall@ 8 (%) on Qwen replay states with source disagreement. Differences and confidence intervals are in percentage points. Intervals use 2,000 paired image-cluster bootstrap resamples of the fixed evaluation set.
Model
Dataset
Two available sources
Three available sources
Calls
Mean (w1,w2)
Calls
Mean (w1,w2,w3)
Qwen
ChartQA
15,101
(0.764,0.236)
12,072
(0.697,0.197,0.105)
TextVQA
12,689
(0.772,0.228)
8,353
(0.700,0.197,0.103)
OCRBench
20,883
(0.758,0.242)
19,394
(0.693,0.199,0.108)
MMT-Bench
44,809
(0.765,0.235)
32,106
(0.689,0.200,0.110)
Pooled
93,482
(0.764,0.236)
71,925
(0.693,0.199,0.108)
Appendix
Table 17: Mean normalized fusion weights in the online ablation, averaged over calls at all candidate-tree positions. Each model uses 200 inputs under two orders. Source indices denote specificity ranks among available rows. Single-source calls have weight one and are omitted.
Dataset
Qwen
LLaVA
P10
P50
P90
ri>1 (%)
P10
P50
P90
ri>1 (%)
MMT-Bench
1.20
1.66
2.20
98.8
0.96
1.53
2.46
88.2
SEED-Bench
1.05
1.24
1.52
96.9
1.09
1.29
1.60
96.9
ScienceQA
1.15
1.66
2.50
98.5
0.99
1.49
2.36
89.4
OCRBench
1.00
1.37
2.97
90.0
0.99
1.66
2.86
88.9
ChartQA
0.89
1.49
3.19
88.4
1.23
1.56
1.97
99.2
Appendix
Table 18: Distributions of paired throughput ratios for Recap on the seven single-turn benchmarks ( 1,000 inputs per dataset and model). P10/P50/P90 report speedup factors; the final column for each model gives the percentage of inputs with strictly higher throughput than paired AR.
Speculative Decoding (SD) has demonstrated its efficacy in accelerating large language model inference, but for Vision-Language Models (VLMs), the diverse input modalities make it difficult for a small trained drafter to achieve consistently high throughput. Self-speculative decoding reuses parts of the target model as a drafter which retains the multi-modal capability of the VLMs, yet its efficiency depends on choosing a suitable draft configuration for each prompt. We propose Single-Peak SPeculative decoding (SP2ec), a training-free bandit algorithm that adapts the draft configuration during generation. SP2ec exploits the single-peak throughput pattern observed in existing self-speculative methods, including SWIFT, KnapSpec, SpecVLM, and our simple cosSim. Our algorithmic design explicitly minimizes a novel notion of wall-time regret for each given query, corresponding to the empirical wall-time latency. Theoretical guarantees on the wall-time regret show that SP2ec can exploit the optimal draft configuration, with negligible exploration cost. Extensive experiments on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B across visual and textual tasks show consistent improvements of up to 11% over the original self-speculative methods, demonstrating the wide applicability of the proposed SP2ec algorithm.
Yuqi Huang, Xingyao Li, Yunlong Hou +3
Department of Mathematics, National University of Singapore, Singapore · Yale University, Center for Algorithms, Data, and Market Design, USA · Department of Electrical and Computer Engineering, National University of Singapore, Singapore
Autoregressive decoding in Large Language Models (LLMs) generates one token per step, causing high inference latency. Speculative decoding (SD) mitigates this through a guess-and-verify strategy, but existing training-free variants face trade-offs: retrieval-based drafts break when no exact match exists, while logits-based drafts lack structural guidance. We propose RACER (Retrieval-Augmented Contextual Rapid Speculative Decoding), a lightweight and training-free method that integrates retrieved exact patterns with logit-driven future cues. This unification supplies both reliable anchors and flexible extrapolation, yielding richer speculative drafts. Experiments on Spec-Bench, HumanEval, and MGSM-ZH demonstrate that RACER consistently accelerates inference, achieving more than 2× speedup over autoregressive decoding, and outperforms prior training-free methods, offering a scalable, plug-and-play solution for efficient LLM decoding. Our source code is available at https://github.com/hkr04/RACER.
Zihong Zhang, Zuchao Li, Lefei Zhang +2
School of Artificial Intelligence, Wuhan University, Wuhan, China · School of Computer Science, Wuhan University, Wuhan, China · School of Information Management, Wuhan University, Wuhan, China +1
Vision-language models (VLMs) achieve strong performance on multimodal tasks but suffer from high inference latency due to large model sizes and long multimodal contexts. Speculative decoding has recently emerged as an effective acceleration technique, yet its behavior in VLMs remains insufficiently understood. We introduce MMSpec, the first benchmark for evaluating speculative decoding in vision-language models. MMSpec contains 600 multimodal samples across six task categories and integrates ten representative speculative decoding algorithms under a unified evaluation framework. Our study reveals three key findings: (1) methods designed for text-only LLMs degrade in multimodal scenarios, (2) vision awareness becomes increasingly important at larger batch sizes, and (3) throughput speedup alone does not reliably reflect latency performance. Motivated by these findings, we propose ViSkip, a plug-and-play speculative decoding method that dynamically adapts speculation to vision tokens and achieves state-of-the-art performance.
Hui Shen, Xin Wang, Ping Zhang +11
University of Michigan · The Ohio State University · Independent +5