Organizations: School of Automation, Beijing Institute of Technology, Beijing, China 100081 · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China 430074 · Distributed Computing Lab, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland 1015
Tree-structured speculative decoding accelerates vision-language models (VLMs) by verifying multiple candidate continuations in parallel. However, long visual contexts increase the cost of speculative computation, making candidate coverage and redundant expansion important considerations beyond tree topology adaptation. We propose Raptor, a training-free framework that uses rejection feedback to jointly regulate candidate coverage and drafting effort. Our key insight is that acceptance length alone cannot distinguish insufficient candidate coverage from excessive deep expansion, and these conditions require different allocation decisions. Raptor therefore extracts complementary rejection-aware signals to characterize speculative progress, frontier coverage, and redundant expansion. These signals jointly guide bounded, momentum-smoothed updates of tree depth, branching width, and drafting budget, together with execution control that limits unnecessary drafting and progressively restores expansion when appropriate. Extensive experiments with two VLMs across seven multimodal benchmarks demonstrate the effectiveness of Raptor, achieving up to 2.39x average speedup over autoregressive decoding.
Figures & tables
Figure 1: Overview of Rejection-aware Raptor. Conventional tree-structured speculative decoding relies on fixed draft tree configurations, where rejected branches are discarded without contributing to future tree construction. Raptor reuses rejection information as feedback, calibrates the next draft tree through an adaptive controller, and dynamically balances candidate coverage and speculative waste across decoding rounds.
Figure 2: Detailed optimization workflow of the Raptor framework. Given multimodal inputs, the draft model constructs speculative trees that are verified by the target model, from which three rejection-aware signals, including acceptance length, frontier gap, and deep waste, are extracted. These signals are converted into adaptive controls to update tree depth, branching width, and drafting budget through momentum-based regulation.
Method
COCO Caps
MME
MM-Vet
SQA
TextVQA
VizWiz
MMMU
Avg.
τ
Spd.
τ
Spd.
τ
Spd.
τ
Spd.
τ
Spd.
τ
Spd.
τ
Spd.
τ
Spd.
LLaVA-1.6
Medusa
0.27
1.02x
0.46
1.20x
0.58
1.74x
0.37
1.09x
0.60
1.39x
0.32
1.04x
0.67
1.53x
0.47
1.29x
+SAGE
0.06
0.91x ↓ 10.8%
0.06
0.90x ↓ 25.0%
0.20
1.39x ↓ 20.1%
0.09
0.96x ↓ 11.9%
0.32
1.14x ↓ 18.0%
0.15
0.96x ↓ 7.7%
0.25
1.13x ↓ 26.1%
0.16
1.06x ↓ 17.8%
+TALON
0.56
1.22x ↑ 19.6%
0.54
1.24x ↑ 3.3%
0.73
2.16x ↑ 24.1%
0.63
1.14x ↑ 4.6%
0.95
1.67x ↑ 20.1%
0.59
1.18x ↑ 13.5%
0.83
1.55x ↑ 1.3%
0.69
1.45x ↑ 12.4%
+Raptor
0.65
1.35x ↑ 32.4 %
0.47
1.36x ↑ 13.3 %
0.86
2.45x ↑ 40.8 %
0.87
1.33x ↑ 22.0 %
1.10
1.81x ↑ 30.2 %
0.75
1.40x ↑ 34.6 %
1.06
1.70x ↑ 11.1 %
0.82
1.63x ↑ 26.4 %
Table 1: Comparison between Raptor and existing adaptive tree strategies across VLMs, speculative decoding frameworks, and multimodal benchmarks, with original speculative decoding configurations included as baselines. Speedups (Spd.) are measured relative to autoregressive decoding, and the highest speedup is highlighted in bold.
Figure 3: Comparison between Raptor (ours) and existing efficient tree adaptation strategies on seven multimodal benchmarks. Speedups over autoregressive decoding are arithmetically averaged across Medusa, EAGLE, and ViSpec. Raptor achieves the highest average speedup on every benchmark for both models.
Table 5
Figure 4: Comparison of generation traces for SAGE, TALON, and Raptor. Each example presents the input image, question, and corresponding decoding traces. Black text denotes accepted draft tokens, green text denotes target-generated tokens, and red strikethrough text denotes rejected draft tokens. In these examples, Raptor adapts candidate exploration using rejection feedback, reducing rejections and corrections during generation.
Figure 5: Per-round decoding behavior under different tree adaptation strategies. Light-blue bars show the number of non-root draft nodes submitted for verification (left axis). The orange solid and red dashed lines indicate accepted draft tokens and actual draft forward calls, respectively (right axis). Raptor achieves longer accepted prefixes on average and requires fewer verification rounds.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Relationship between rejection-aware signals and subsequent proxy reference configurations. (a)–(c) show trends in normalized reference depth, branching width, and node budget conditioned on each signal. (d) reports correlations between the signals and reference configuration dimensions. FG, DW, and AL denote frontier gap, deep waste, and acceptance length, respectively.
Figure 7: Variation in proxy reference node budgets across combinations of frontier gap and deep waste.
Speculative decoding accelerates vision-language model (VLM) inference without altering the output distribution, yet two factors limit its gains. Existing drafters either lack the image evidence required for visually grounded tokens or pay to condition on every visual token, although a short draft block typically depends on only a few image regions. Moreover, the value of verifying additional candidates varies with GPU load, so no fixed verification structure is efficient across serving conditions. We present Uni-LaSpec, a block-parallel speculative decoding framework that addresses both factors. To improve draft quality, Uni-LaSpec reuses the verifier's own attention: the query retained from the preceding verification ranks the cached, int8-quantized visual keys, and a cross-layer vote selects a few visual tokens that the drafter fuses through gated cross-attention. Acceptance-aware training further weights each draft position by its contribution to the accepted prefix. To allocate verification, a marginal-cost criterion selects the structure per request and per round: draft trees exploit idle verifier capacity under light load, whereas a dynamic head truncates linear drafts under heavy load. Across six VLMs from three model families and 17 text and multimodal benchmarks, selecting eight visual tokens retains at least 98.6% of the acceptance obtained from the full image at sub-millisecond selection cost. With greedy decoding at concurrency one, Uni-LaSpec achieves up to 5.11x text and 3.24x multimodal throughput over autoregressive decoding. Across six load levels and three GPU types, it outperforms autoregressive decoding in every setting and matches or exceeds the best fixed verification strategy in 123 of 144 settings; under bursty arrivals, it reduces mean end-to-end latency by 47.6% relative to linear verification.
Hong Nie, Zhangchi Gao, Yuqian Liang +4
University of Science and Technology of China · Unisound AI Technology Co., Ltd.
Speculative decoding accelerates large language model inference through a draft-then-verify paradigm. Building on this, tree-structured methods improve inference by organizing proposals into multiple candidate paths, increasing the accepted length. However, existing tree-structured methods use a single drafter for all drafting steps, creating a dilemma: a smaller drafter is fast but yields lower-quality trees, whereas a larger drafter improves tree quality but suffers from high latency. To address this, we propose TreeGraft, a multi-drafter framework in which drafters of different costs jointly construct a shared draft tree. TreeGraft uses the stronger drafter to rescore candidates by updating scores assigned by the weaker drafter, reselect grafting positions, and recover promising paths left unexplored. It also integrates stronger drafter expansions non-destructively, preserving existing branches that may still be accepted by the target model. Together, these designs improve the quality of the shared draft tree. To control the drafting cost, TreeGraft introduces a lightweight scheduler distilled from an offline value system to decide when to call the stronger drafter. Across 10 model pairs and 6 benchmarks, TreeGraft outperforms the better of the two fixed single-drafter endpoint strategies by 15.1% on average, reaching a maximum gain of 26.6%. Our code is available at https://github.com/fjm9933/TreeGraft.
Jiaming Fan, Daming Cao, Canchen Huang +6
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Southeast University · Ant Group +1
Speculative Decoding (SD) has demonstrated its efficacy in accelerating large language model inference, but for Vision-Language Models (VLMs), the diverse input modalities make it difficult for a small trained drafter to achieve consistently high throughput. Self-speculative decoding reuses parts of the target model as a drafter which retains the multi-modal capability of the VLMs, yet its efficiency depends on choosing a suitable draft configuration for each prompt. We propose Single-Peak SPeculative decoding (SP2ec), a training-free bandit algorithm that adapts the draft configuration during generation. SP2ec exploits the single-peak throughput pattern observed in existing self-speculative methods, including SWIFT, KnapSpec, SpecVLM, and our simple cosSim. Our algorithmic design explicitly minimizes a novel notion of wall-time regret for each given query, corresponding to the empirical wall-time latency. Theoretical guarantees on the wall-time regret show that SP2ec can exploit the optimal draft configuration, with negligible exploration cost. Extensive experiments on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B across visual and textual tasks show consistent improvements of up to 11% over the original self-speculative methods, demonstrating the wide applicability of the proposed SP2ec algorithm.
Yuqi Huang, Xingyao Li, Yunlong Hou +3
Department of Mathematics, National University of Singapore, Singapore · Yale University, Center for Algorithms, Data, and Market Design, USA · Department of Electrical and Computer Engineering, National University of Singapore, Singapore