cs.CVNov 19, 2026

Raptor: Rejection-aware Adaptive Tree based on Speculative Decoding

Authors: Qishu Hu, Haozhao Wang, Zicong Hong, Yufeng Zhan, Yuanqing Xia

Organizations: School of Automation, Beijing Institute of Technology, Beijing, China 100081 · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China 430074 · Distributed Computing Lab, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland 1015

Abstract

Tree-structured speculative decoding accelerates vision-language models (VLMs) by verifying multiple candidate continuations in parallel. However, long visual contexts increase the cost of speculative computation, making candidate coverage and redundant expansion important considerations beyond tree topology adaptation. We propose Raptor, a training-free framework that uses rejection feedback to jointly regulate candidate coverage and drafting effort. Our key insight is that acceptance length alone cannot distinguish insufficient candidate coverage from excessive deep expansion, and these conditions require different allocation decisions. Raptor therefore extracts complementary rejection-aware signals to characterize speculative progress, frontier coverage, and redundant expansion. These signals jointly guide bounded, momentum-smoothed updates of tree depth, branching width, and drafting budget, together with execution control that limits unnecessary drafting and progressively restores expansion when appropriate. Extensive experiments with two VLMs across seven multimodal benchmarks demonstrate the effectiveness of Raptor, achieving up to 2.39x average speedup over autoregressive decoding.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Seeing What Matters, Verifying What Pays: Load-Adaptive Speculative Decoding for Vision-Language Models

    Nov 1, 2026Hong Nie, Zhangchi Gao, Yuqian Liang +4Efficient VLM InferenceSpeculative Decoding

  2. TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding

    May 28, 2026Jiaming Fan, Daming Cao, Canchen Huang +6Speculative DecodingLLM Inference Acceleration

  3. SP2^2ec: Adaptive Self-Speculative Decoding for Vision-Language Models

    Oct 13, 2026Yuqi Huang, Xingyao Li, Yunlong Hou +3Speculative DecodingEfficient VLM Inference