Vision-language models (VLMs) predominantly rely on autoregressive decoding, which generates tokens one at a time and fundamentally limits inference throughput. This limitation is especially acute in physical AI scenarios such as robotics and autonomous driving, where VLMs are deployed on edge devices at batch size one, making AR decoding memory-bandwidth-bound and leaving hardware parallelism underutilized. While block-wise discrete diffusion has shown promise for parallel text generation, extending it to VLMs remains challenging due to the need to jointly handle continuous visual representations and discrete text tokens while preserving pretrained multimodal capabilities. We present Fast-dVLM, a block-diffusion-based VLM that enables KV-cache-compatible parallel decoding and speculative block decoding for inference acceleration. We systematically compare two AR-to-diffusion conversion strategies: a two-stage approach that first adapts the LLM backbone with text-only diffusion fine-tuning before multimodal training, and a direct approach that converts the full AR VLM in one stage. Under comparable training budgets, direct conversion proves substantially more efficient by leveraging the already multimodally aligned VLM; we therefore adopt it as our recommended recipe. We introduce a suite of multimodal diffusion adaptations, block size annealing, causal context attention, auto-truncation masking, and vision efficient concatenation, that collectively enable effective block diffusion in the VLM setting. Extensive experiments across 11 multimodal benchmarks show Fast-dVLM matches its autoregressive counterpart in generation quality. With SGLang integration and FP8 quantization, Fast-dVLM achieves over 6x end-to-end inference speedup over the AR baseline.
Figures & tables
Figure 1: Overview of our Fast-dVLM. (a) Fast-dVLM achieves comparable accuracy to AR VLM baselines on MMMU-Pro-V with a substantial speedup. (b) Benchmark comparison against the Qwen2.5-VL-3B backbone, showing near-lossless accuracy across diverse tasks. (c) Our combined approach achieves up to 6.18 × end-to-end speedup compared to the AR baseline. All throughput measurements are conducted on a single NVIDIA H100 GPU.
Figure 2: Two AR-to-diffusion conversion strategies. (a) The two-stage strategy first converts the LLM backbone via text-only diffusion fine-tuning (Stage 1), then fine-tunes all components on multimodal data (Stage 2). (b) The direct path converts the full VLM in a single multimodal stage.
Figure 3: Training architecture and attention mask for [wt;x] with block size B^=2 . The noisy stream wt contains only text tokens; vision tokens appear exclusively in the clean stream x (vision-efficient concatenation). MN2N : block-diagonal attention for parallel denoising. MN2C : noisy tokens attend to preceding clean context including vision tokens. MC2C : token-level causal attention on the clean stream, enabling joint AR loss training and AR decoding at inference.
Short-answer Benchmarks
Model
AI2D
ChartQA
DocVQA
GQA
MMBench
MMMU
POPE
Autoregressive Vision-Language Models
VILA-1.5-3B
58.0
53.0
44.3
61.4
60.5
31.8
86.8
MiniCPM-V-2 (3B)
65.0
59.2
69.8
51.7
66.3
37.9
86.5
Intern-VL-2.5-4B
81.3
77.8
91.1
61.0
80.7
50.0
89.3
Qwen2.5-VL-3B
80.8
84.0
93.1
59.0
76.9
47.3
86.2
Table 1: Benchmark performance comparison (Part 1: short-answer benchmarks). Models are grouped into autoregressive (AR) and diffusion categories. Among diffusion models, best and 2nd best results are highlighted. MDM = masked diffusion model decoding; spec. = speculative decoding.
Short-answer Benchmarks
Long-answer
Model
RealWorldQA
SEEDBench2+
TextVQA
Avg
MMMU-Pro-V
Tokens/NFE
Autoregressive Vision-Language Models
VILA-1.5-3B
53.2
41.2
58.2
54.8
6.1
1.00
MiniCPM-V-2 (3B)
56.3
52.5
74.4
62.0
10.3
1.00
Intern-VL-2.5-4B
64.6
67.0
78.8
74.2
24.6
1.00
Qwen2.5-VL-3B
65.1
68.6
79.1
74.0
26.3
1.00
Table 2: Benchmark performance comparison (Part 2). Tokens/NFE measures average tokens decoded per forward pass. Among diffusion models, best and 2nd best results are highlighted. MDM = masked diffusion model decoding; spec. = speculative decoding.
Figure 4: Direct vs. two-stage adaptation across 10 benchmarks. Each axis is independently scaled.
Setting
MMBench
MMMU
POPE
MMMU-Pro-V
RealWorldQA
SeedBench2+
Avg
Full recipe
72.4
43.0
85.1
15.1
61.1
66.9
57.3
w/o causal context
58.5 (-19.2%)
29.9 (-30.5%)
71.1 (-16.5%)
6.2 (-58.9%)
60.0 (-1.8%)
40.5 (-39.5%)
44.4 (-22.5%)
w/o annealing block size
68.6 (-5.2%)
43.4 (+0.9%)
81.4 (-4.3%)
10.2 (-32.5%)
58.4 (-4.4%)
66.8 (-0.1%)
54.8 (-4.4%)
w/o auto-truncation
68.4 (-5.5%)
36.8 (-14.4%)
84.3 (-0.9%)
13.5 (-10.6%)
61.0 (-0.2%)
67.1 (+0.3%)
55.2 (-3.7%)
Table 3: Ablation study on training recipe. Each row removes one component from the full recipe. Parenthesized values show relative change: red for drops, green for gains.
Figure 5: Effect of threshold τ on MMMU-Pro CoT accuracy (left axis) and tokens per step (right axis).
Figure 6: Speculative decoding throughput across block sizes, comparing linear and quadratic variants. (a) Tokens per NFE increases with block size for both variants. (b) Wall-clock TPS peaks at block size 16 for both variants; the quadratic variant’s O(B2) cost becomes dominant at larger block sizes.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
MMMU-Pro-V
TPS
SpeedUp
AR baseline
26.3
56.7
1.00 ×
Fast-dVLM (MDM, τ=0.9 )
21.4
82.2
1.45 ×
+ Spec. decoding (linear)
24.6
112.7
1.98 ×
+ SGLang serving
24.1
319.0
5.63 ×
+ SmoothQuant-W8A8 (FP8)
23.8
350.3
6.18 ×
Appendix
Table 4: Inference acceleration on MMMU-Pro-V. Each row progressively adds one optimization on top of the previous. SpeedUp is relative to the AR baseline.
Figure 7: Qualitative comparison on an MMMU-Pro-V math reasoning problem. Both the AR baseline (Qwen2.5-VL-3B) and Fast-dVLM (speculative decoding) arrive at the correct answer, while Fast-dVLM achieves 1.6× faster decoding speed (77.7 vs. 47.4 tokens/s).
Figure 8: Additional qualitative examples spanning art style recognition, celebrity identification, and chart QA. Fast-dVLM produces accurate responses while maintaining high throughput.
Figure 9: Qualitative examples of Fast-dVLM (speculative decoding) on embodied and physical AI tasks: autonomous driving scene understanding and robotic manipulation instruction. For each example, we report the generated response length (Gen. Len), decoding throughput (Tok./sec), and tokens per decoding step (Tok./step). Fast-dVLM produces detailed, step-by-step reasoning for both tasks while maintaining high throughput ( ∼ 73 tokens/s) and a Tok./step ratio above 1.68.
Autoregressive vision-language models (VLMs) deliver strong multimodal capability, but their token-by-token decoding imposes a fundamental inference bottleneck. Diffusion VLMs offer a more parallel decoding paradigm, yet directly converting a pretrained autoregressive VLM into a large-block diffusion VLM (dVLM) often leads to substantial quality degradation. In this work, we present BARD, a simple and effective bridging framework that converts a pretrained autoregressive VLM into a same-architecture, decoding-efficient dVLM. Our approach combines progressive supervised block merging, which gradually enlarges the decoding block size, with stage-wise intra-dVLM distillation from a fixed small-block diffusion anchor to recover performance lost at larger blocks. We further incorporate a mixed noise scheduler to improve robustness and token revision during denoising, and memory-friendly training to enable efficient training on long multimodal sequences. A key empirical finding is that direct autoregressive-to-diffusion distillation is poorly aligned and can even hurt performance, whereas distillation within the diffusion regime is consistently effective. Experimental results show that, with ≤ 4.4M data, BARD-VL transfers strong multimodal capability from Qwen3-VL to a large-block dVLM. Remarkably, BARD-VL establishes a new SOTA among comparable-scale open dVLMs on our evaluation suite at both 4B and 8B scales. At the same time, BARD-VL achieves up to 3× decoding throughput speedup compared to the source model. Code is available at https://github.com/fudan-generative-vision/Bard-VL.
Baoyou Chen, Hanchen Xia, Peng Tu +5
Shanghai Academy of AI for Science · Fudan University · Shanghai Innovation Institute +1
Modern Vision-Language Models (VLMs) can solve a wide range of tasks requiring visual reasoning. In real-world scenarios, desirable properties for VLMs include fast inference and controllable generation (e.g., constraining outputs to adhere to a desired format). However, existing autoregressive (AR) VLMs like LLaVA struggle in these aspects. Discrete diffusion models (DMs) offer a promising alternative, enabling parallel decoding for faster inference and bidirectional context for controllable generation through text-infilling. While effective in language-only settings, DMs' potential for multimodal tasks is underexplored. We introduce LaViDa, a family of VLMs built on DMs. We build LaViDa by equipping DMs with a vision encoder and jointly fine-tune the combined parts for multimodal instruction following. To address challenges encountered, LaViDa incorporates novel techniques such as complementary masking for effective training, prefix KV cache for efficient inference, and timestep shifting for high-quality sampling. Experiments show that LaViDa achieves competitive or superior performance to AR VLMs on multi-modal benchmarks such as MMMU, while offering unique advantages of DMs, including flexible speed-quality tradeoff, controllability, and bidirectional reasoning. On COCO captioning, LaViDa surpasses Open-LLaVa-Next-8B by +4.1 CIDEr with 1.92x speedup. On bidirectional tasks, it achieves +59% improvement on Constrained Poem Completion. These results demonstrate LaViDa as a strong alternative to AR VLMs. Code and models will be released in the camera-ready version.
Speculative decoding (SD) has proven to be an effective technique for accelerating autoregressive generation in large language models (LLMs) however, its application to vision-language models (VLMs) remains relatively unexplored. We propose~DREAM-S, a novel SD framework designed specifically for fast and efficient decoding in VLMs. DREAM-S leverages a neural architecture search (NAS) framework with target-aware supernet training to automatically identify both the optimal interaction strategy between the draft and target models, and the most suitable draft model architecture for the underlying hardware implementation platform. DREAM-S additionally incorporates adaptive intermediate feature distillation, guided by attention entropy, to enable efficient draft training. Experiments on a range of well-established VLMs show that DREAM-S achieves up to a 3.85× speedup compared to standard decoding approaches and significantly outperforms existing SD baselines. The code is publicly available at: https://github.com/SAI-Lab-NYU/DREAM-S .