Raptor: Rejection-aware Adaptive Tree based on Speculative Decoding
Organizations: School of Automation, Beijing Institute of Technology, Beijing, China 100081 · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China 430074 · Distributed Computing Lab, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland 1015
Abstract
Tree-structured speculative decoding accelerates vision-language models (VLMs) by verifying multiple candidate continuations in parallel. However, long visual contexts increase the cost of speculative computation, making candidate coverage and redundant expansion important considerations beyond tree topology adaptation. We propose Raptor, a training-free framework that uses rejection feedback to jointly regulate candidate coverage and drafting effort. Our key insight is that acceptance length alone cannot distinguish insufficient candidate coverage from excessive deep expansion, and these conditions require different allocation decisions. Raptor therefore extracts complementary rejection-aware signals to characterize speculative progress, frontier coverage, and redundant expansion. These signals jointly guide bounded, momentum-smoothed updates of tree depth, branching width, and drafting budget, together with execution control that limits unnecessary drafting and progressively restores expansion when appropriate. Extensive experiments with two VLMs across seven multimodal benchmarks demonstrate the effectiveness of Raptor, achieving up to 2.39x average speedup over autoregressive decoding.
Figures & tables
| Method | COCO Caps | MME | MM-Vet | SQA | TextVQA | VizWiz | MMMU | Avg. | ||||||||
| Spd. | Spd. | Spd. | Spd. | Spd. | Spd. | Spd. | Spd. | |||||||||
| LLaVA-1.6 | ||||||||||||||||
| Medusa | 0.27 | 1.02x | 0.46 | 1.20x | 0.58 | 1.74x | 0.37 | 1.09x | 0.60 | 1.39x | 0.32 | 1.04x | 0.67 | 1.53x | 0.47 | 1.29x |
| +SAGE | 0.06 | 0.91x 10.8% | 0.06 | 0.90x 25.0% | 0.20 | 1.39x 20.1% | 0.09 | 0.96x 11.9% | 0.32 | 1.14x 18.0% | 0.15 | 0.96x 7.7% | 0.25 | 1.13x 26.1% | 0.16 | 1.06x 17.8% |
| +TALON | 0.56 | 1.22x 19.6% | 0.54 | 1.24x 3.3% | 0.73 | 2.16x 24.1% | 0.63 | 1.14x 4.6% | 0.95 | 1.67x 20.1% | 0.59 | 1.18x 13.5% | 0.83 | 1.55x 1.3% | 0.69 | 1.45x 12.4% |
| +Raptor | 0.65 | 1.35x 32.4 % | 0.47 | 1.36x 13.3 % | 0.86 | 2.45x 40.8 % | 0.87 | 1.33x 22.0 % | 1.10 | 1.81x 30.2 % | 0.75 | 1.40x 34.6 % | 1.06 | 1.70x 11.1 % | 0.82 | 1.63x 26.4 % |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Efficient VLM Inference
Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.
See all 530 papers