RECAP: Context-Aware Prediction Recycling with Request-Local and Persistent Memories for Multimodal Speculative Decoding
Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences
Abstract
Recycling a target model's own predictions offers a training-free source of proposals for speculative decoding. In vision-language models, however, existing recycling indexes predictions by the last token alone, conflating predictions computed under different images and prefixes; we find that such reuse fails most often at image-sensitive decoding states. Longer keys reduce this aliasing but match fewer states. We introduce \method, which stores target predictions under one-, two-, and three-token keys: contextual memories live only for the current request and are thus implicitly conditioned on its image, while token-level memory persists across requests to retain coverage. At each lookup, entropy-confidence weighting fuses the retrieved predictions according to their key specificity and concentration. Best-first search composes the fused transitions into a candidate tree, supplemented by suffix-matched continuations, and one target pass verifies the tree under greedy decoding. Without training or an auxiliary draft model, \method\ achieves mean speedups over autoregressive decoding of and across seven single-turn benchmarks on Qwen2.5-VL-7B and LLaVA-1.6-7B, exceeding the strongest training-free baseline, Token Recycling ( and ). Our implementation is available in https://github.com/Senmo996/RECAP-official-code.
Figures & tables
| Models | Methods | MMT | SEED | SciQA | OCR | ChartQA | MathVista | TextVQA | Average | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-VL 7B-Instruct | PLD | 1.16 | 1.19 | 0.89 | 1.05 | 1.24 | 1.13 | 1.22 | 1.20 | 1.14 | 1.25 | 1.49 | 1.22 | 0.97 | 1.09 | 1.16 | 1.16 |
| Lookahead | 1.15 | 1.60 | 0.93 | 1.35 | 1.44 | 2.14 | 1.11 | 1.62 | 1.10 | 1.62 | 1.20 | 1.60 | 0.94 | 1.26 | 1.12 | 1.60 | |
| Tree Recycling | 1.20 | 1.47 | 1.07 | 1.49 | 1.23 | 1.54 | 1.29 | 1.68 | 1.27 | 1.72 | 1.41 | 1.70 | 1.06 | 1.41 | 1.22 | 1.57 | |
| Token Recycling | 1.59 | 2.57 | 1.25 | 2.58 | 1.60 | 2.60 | 1.51 | 2.82 | 1.54 | 2.89 | 1.86 | 2.84 | 1.21 | 2.63 | 1.51 | 2.71 | |
| SAM | 1.11 | 1.24 | 1.06 | 1.34 | 1.15 | 1.36 | 1.20 | 1.33 | 1.23 | 1.57 | 1.21 | 1.38 | 1.05 | 1.26 | 1.14 | 1.35 | |
| Models | Methods | General VQA | Text VQA | Captioning | Charts | Reasoning | Multi-turn | Overall | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-VL 7B-Instruct | PLD | 1.06 | 1.05 | 1.21 | 1.13 | 1.14 | 1.15 | 1.20 | 1.31 | 1.28 | 1.34 | 1.34 | 1.27 | 1.23 | 1.22 |
| Lookahead | 1.11 | 1.37 | 1.04 | 1.29 | 1.44 | 1.64 | 1.21 | 1.77 | 1.91 | 2.28 | 1.39 | 1.80 | 1.36 | 1.71 | |
| Tree Recycling | 1.12 | 1.40 | 1.10 | 1.32 | 1.22 | 1.37 | 1.21 | 1.64 | 1.38 | 1.66 | 1.25 | 1.56 | 1.22 | 1.51 | |
| Token Recycling | 1.51 | 2.57 | 1.31 | 2.49 | 1.80 | 2.51 | 1.38 | 2.60 | 1.77 | 2.64 | 1.57 | 2.60 | 1.56 | 2.57 | |
| SAM | 1.05 | 1.14 | 1.13 | 1.13 | 1.03 | 1.07 | 1.14 | 1.54 | 1.18 | 1.31 | 1.08 | 1.23 | 1.10 | 1.24 | |
| Model | Method | Params | GiB | AR | ||
| DREAM | 607M | 1.13 | 3.68 | 1.26 | 41.9 | |
| ViSpec | 451M | 0.84 | 3.74 | 1.05 | 61.2 | |
| LLaVA | Recap | 0 | 0 | 2.92 | 1.56 | 29.9 |
| ViSpec | 890M | 1.66 | 3.76 | 1.18 | 57.0 | |
| Qwen | Recap | 0 | 0 | 3.02 | 1.66 | 39.4 |
| Comparison | Gain (%, ) |
|---|---|
| Candidate search | |
| Best-first vs. fixed expansion | |
| Memory design | |
| Context backoff vs. unigram | |
| Uniform fusion vs. backoff | |
| Specificity priors vs. uniform fusion | |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Coverage (%, ) | Recall@ (%, ) | ||||
|---|---|---|---|---|---|---|
| ChartQA | 44.5 | 23.2 | 14.8 | 64.5 | 81.7 | 86.6 |
| TextVQA | 26.3 | 7.4 | 3.4 | 52.5 | 73.5 | 78.3 |
| OCRBench | 54.9 | 30.4 | 18.6 | 63.6 | 84.2 | 90.6 |
| MMT-Bench | 45.6 | 19.7 | 11.2 | 56.9 | 77.9 | 84.4 |
| Policy | Matched continuation length ( ) | |||
|---|---|---|---|---|
| ChartQA | TextVQA | OCR | MMT | |
| 0.516 | 0.224 | 0.632 | 0.444 | |
| backoff | 0.709 | 0.253 | 1.135 | 0.600 |
| backoff | 0.778 | 0.252 | 1.128 | 0.638 |
| Uniform fusion | 0.737 | 0.253 | 1.041 | 0.629 |
| Prior-weighted fusion | 0.751 | 0.253 | 1.066 | 0.639 |
| Dataset | Offline matched continuation length | Online fusion + suffix / fusion | |||
| Fusion | + Suffix | Difference | Gain (%, ) | : before after | |
| ChartQA | 1.123 | 1.313 | 0.190 | ||
| TextVQA | 0.355 | 0.413 | 0.059 | ||
| OCRBench | 1.417 | 1.538 | 0.120 | ||
| MMT-Bench | 0.750 | 0.866 | 0.116 | ||
| Dataset | Throughput gain (%, ) | |
|---|---|---|
| Blocked | Alternating | |
| ChartQA | ||
| TextVQA | ||
| OCRBench | ||
| MMT-Bench | ||
| Qwen | LLaVA | |||
|---|---|---|---|---|
| Stage | Wall | CPU | Wall | CPU |
| Suffix lookup | ||||
| Lookup, fusion, search | ||||
| Packing | ||||
| Memory refresh | ||||
| Suffix index update | ||||
| Model | Cap | Mean nodes | Prep. ms | ms | Host MiB | GPU GiB |
|---|---|---|---|---|---|---|
| Qwen | 31 | 26.5 | 0.591 | 0.722 | 5.79 | 19.32 |
| Qwen | 63 | 55.2 | 0.907 | 1.227 | 7.88 | 19.32 |
| Qwen | 127 | 113.4 | 2.202 | 2.617 | 8.74 | 19.32 |
| LLaVA | 31 | 29.1 | 0.721 | 0.882 | 7.46 | 19.98 |
| LLaVA | 63 | 59.2 | 1.121 | 1.385 | 8.70 | 20.01 |
| LLaVA | 127 | 121.0 | 2.485 | 2.729 | 12.68 | 20.16 |
| Model | Host MiB | Lookup ns | |
|---|---|---|---|
| Qwen | |||
| LLaVA | |||
| Variant | ( , ) | ( ) |
|---|---|---|
| Recap (lexical) | 1.63 | 2.67 |
| Recap + semantic fallback | 1.61 | 2.78 |
| Recap + recency fallback | 1.61 | 2.75 |
| Variant | Qwen | LLaVA | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ChartQA | TextVQA | OCR | MMT | Mean | ChartQA | TextVQA | OCR | MMT | Mean | |||
| Recap | ||||||||||||
| w/o confidence | ||||||||||||
| w/o prior | ||||||||||||
| Uniform | ||||||||||||
| Confidence gate | ||||||||||||
| Variant | ChartQA | TextVQA | OCR | MMT |
|---|---|---|---|---|
| Recap | ||||
| w/o confidence | ||||
| w/o prior | ||||
| Uniform | ||||
| Confidence gate | ||||
| Specificity backoff |
| Model | Method | P50 | P90 | P95 | P99 |
|---|---|---|---|---|---|
| Qwen | AR | 2.70 | 8.44 | 11.63 | 15.07 |
| Qwen | Recap | 1.52 | 4.46 | 6.08 | 7.56 |
| LLaVA | AR | 4.02 | 9.42 | 14.27 | 18.43 |
| LLaVA | Recap | 2.38 | 5.45 | 8.25 | 10.67 |
| Dataset | States | Recap | Confidence gate | Difference | 95% CI |
|---|---|---|---|---|---|
| ChartQA | 427 | 84.31 | 66.98 | ||
| TextVQA | 52 | 75.00 | 65.38 | ||
| OCRBench | 857 | 62.43 | 46.91 | ||
| MMT-Bench | 510 | 77.65 | 62.55 |
| Model | Dataset | Two available sources | Three available sources | ||
|---|---|---|---|---|---|
| Calls | Mean | Calls | Mean | ||
| Qwen | ChartQA | 15,101 | 12,072 | ||
| TextVQA | 12,689 | 8,353 | |||
| OCRBench | 20,883 | 19,394 | |||
| MMT-Bench | 44,809 | 32,106 | |||
| Pooled | 93,482 | 71,925 | |||
| Dataset | Qwen | LLaVA | ||||||
|---|---|---|---|---|---|---|---|---|
| P10 | P50 | P90 | (%) | P10 | P50 | P90 | (%) | |
| MMT-Bench | 1.20 | 1.66 | 2.20 | 98.8 | 0.96 | 1.53 | 2.46 | 88.2 |
| SEED-Bench | 1.05 | 1.24 | 1.52 | 96.9 | 1.09 | 1.29 | 1.60 | 96.9 |
| ScienceQA | 1.15 | 1.66 | 2.50 | 98.5 | 0.99 | 1.49 | 2.36 | 89.4 |
| OCRBench | 1.00 | 1.37 | 2.97 | 90.0 | 0.99 | 1.66 | 2.86 | 88.9 |
| ChartQA | 0.89 | 1.49 | 3.19 | 88.4 | 1.23 | 1.56 | 1.97 | 99.2 |
Efficient VLM Inference
Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.
See all 509 papers