cs.CVMay 20, 2025

Speculative Decoding Reimagined for Multimodal Large Language Models

Authors: Luxi Lin, Zhihang Lin, Zhanpeng Zeng, Rongrong Ji

Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Shanghai Innovation Institute

Abstract

Multimodal Large Language Models (MLLMs) deliver strong performance across vision-language tasks but suffer from high autoregressive decoding latency. Although speculative decoding can accelerate Large Language Models (LLMs) without changing their output distribution, directly applying existing approaches to MLLMs yields limited speedups because text and visual tokens exhibit distinct characteristics. We present Multimodal Speculative Decoding (MSD), a framework specifically designed for efficient MLLM inference. MSD is motivated by two key observations: text and visual tokens should be handled separately during drafting, and an effective draft model requires both strong language modeling and visual perception capabilities. Accordingly, MSD decouples text and visual token processing in the draft model and applies token-specific strategies to improve drafting efficiency. We further introduce a two-stage training strategy. The draft model is first trained on text-only instruction-tuning data to strengthen its language modeling ability and is then progressively exposed to multimodal data to develop visual perception while preserving its language competence. Extensive experiments across multimodal benchmarks demonstrate that MSD achieves up to 2.29×2.29\times inference speedup for LLaVA-1.5-7B and 2.46×2.46\times for LLaVA-1.5-13B without sacrificing generation quality. These results establish MSD as an effective and general approach for accelerating multimodal autoregressive generation. Our code is available at https://github.com/Lyn-Lucy/MSD

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Nov 13, 2026

MMFlash: Faster Speculative Decoding for Vision-Language Models

Speculative decoding accelerates generation by letting a lightweight drafter propose tokens for parallel verification by a target model. Diffusion drafters such as DFlash propose an entire token block in a single forward pass, but mainly focus on text-only tasks, leaving multimodal drafting underexplored. To study this setting, we first introduce a metric of token-level visual dependence based on the KL divergence between next-token distributions with and without visual input. We find that acceptance rates decrease from 46% in the lowest visual-dependence quartile to 22% in the highest, with this disparity persisting after multimodal training. Motivated by this observation, we propose MMFlash, a simple and effective framework that adaptively upweights visually dependent tokens that the drafter predicts poorly. For long video inputs, MMFlash further employs sparse attention to reduce redundant attention computation and improve drafting quality. Across various multimodal benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58×3.58\times, outperforming EAGLE-3 and DFlash by up to 195.4% and 17.1%, respectively.
Mar 16, 2026

MMSpec: Benchmarking Speculative Decoding for Vision-Language Models

Vision-language models (VLMs) achieve strong performance on multimodal tasks but suffer from high inference latency due to large model sizes and long multimodal contexts. Speculative decoding has recently emerged as an effective acceleration technique, yet its behavior in VLMs remains insufficiently understood. We introduce MMSpec, the first benchmark for evaluating speculative decoding in vision-language models. MMSpec contains 600 multimodal samples across six task categories and integrates ten representative speculative decoding algorithms under a unified evaluation framework. Our study reveals three key findings: (1) methods designed for text-only LLMs degrade in multimodal scenarios, (2) vision awareness becomes increasingly important at larger batch sizes, and (3) throughput speedup alone does not reliably reflect latency performance. Motivated by these findings, we propose ViSkip, a plug-and-play speculative decoding method that dynamically adapts speculation to vision tokens and achieves state-of-the-art performance.
May 30, 2026

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation

Speculative decoding (SD) has proven to be an effective technique for accelerating autoregressive generation in large language models (LLMs) however, its application to vision-language models (VLMs) remains relatively unexplored. We propose~DREAM-S, a novel SD framework designed specifically for fast and efficient decoding in VLMs. DREAM-S leverages a neural architecture search (NAS) framework with target-aware supernet training to automatically identify both the optimal interaction strategy between the draft and target models, and the most suitable draft model architecture for the underlying hardware implementation platform. DREAM-S additionally incorporates adaptive intermediate feature distillation, guided by attention entropy, to enable efficient draft training. Experiments on a range of well-established VLMs show that DREAM-S achieves up to a 3.85×3.85\times speedup compared to standard decoding approaches and significantly outperforms existing SD baselines. The code is publicly available at: https://github.com/SAI-Lab-NYU/DREAM-S .