ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding

Quan Kong; Yuhao Shen; Yicheng Ji; Huan Li; Cong Wang

cs.CVMar 20, 2026

ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding

Authors: Quan Kong, Yuhao Shen, Yicheng Ji, Huan Li, Cong Wang

Organizations: Zhejiang University, Hangzhou, China

Abstract

Although current Video-LLMs achieve impressive performance in video understanding tasks, their autoregressive decoding efficiency remains constrained by the massive number of video tokens. Visual token pruning can partially ease this bottleneck, yet existing approaches still suffer from information loss and yield only modest acceleration in decoding. In this paper, we propose ParallelVLM, a training-free draft-then-verify speculative decoding framework that overcomes both mutual waiting and limited speedup-ratio problems between draft and target models in long-video settings. ParallelVLM features two parallelized stages that maximize hardware utilization and incorporate an Unbiased Verifier-Guided Pruning strategy to better align the draft and target models by eliminating the positional bias in attention-guided pruning. Extensive experiments demonstrate that ParallelVLM effectively expands the draft window by $1.6\sim1.8\times$ with high accepted lengths, and accelerates various video understanding benchmarks by 3.36 $\times$ on LLaVA-Onevision-72B and 2.42 $\times$ on Qwen2.5-VL-32B compared with vanilla autoregressive decoding.

Figures & tables

Figure 2: Comparison of different training-free speculative decoding frameworks for Video-LLMs. (a) Vanilla SD: sequential in both prefilling and decoding stages, achieving speedup only with the draft-then-verify design. (b) SpecVLM [ 17 ] : inherits vanilla SD and enables faster draft model execution via attention-guided pruning. (c) ParallelVLM: adopts a parallel pipeline to mitigate idling time in vanilla SD, co-designed with UV-Prune strategy to expand the draft window size meanwhile preserving draft/target alignment.

Figure 3: Scaling of (a) Prefilling Latency and (b) Decoding Time with increasing number of video tokens. Experiments are conducted on LLaVA-OV 7B draft model and 72B target model.

Figure 5: The proposed ParallelVLM consists of Parallel Prefilling (PP), Parallel Decoding (PD) and the Unbiased Verifier-guided Pruning (UV-Prune). (a) During PP stage: draft model prefilling and pruning executes in parallel with target model prefilling, where UV-Prune transfers salient alignment semantics from the target model to the draft model without “positional bias”. (b) During PD stage: the reduced draft decoding time $T_{q}$ enables an enlarged window size $\gamma$ in parallel with verification, meanwhile maintaining high acceptance rates $\tau$ .

Models

Methods

VideoDetailCaption

VideoMME

MVBench

MVLU

LongVideoBench

Avg.

M

Speedup

M

Speedup

M

Speedup

M

Speedup

M

Speedup

LLaVA-OV (0.5B & 7B)

Vanilla SD

4.26

1.08

\times

4.34

1.10

\times

4.18

1.05

\times

4.22

1.06

\times

4.32

1.01

\times

1.06

\times

OnSD

3.73

1.40

\times

3.67

1.38

\times

3.78

1.32

\times

3.56

1.29

\times

3.76

1.36

\times

1.35

\times

SD-Tree

5.08

1.28

\times

5.05

1.28

\times

4.97

1.25

\times

4.90

1.24

\times

5.12

1.26

\times

1.26

\times

SpecVLM

4.89

1.83

\times

4.83

1.82

\times

4.76

1.77

\times

4.71

1.76

\times

4.92

1.86

\times

1.81

\times

ParallelVLM

8.47

2.12

\times

7.86

2.06

\times

8.64

2.15

\times

8.24

2.14

\times

8.35

2.10

\times

2.11

\times

Table 1: Comparison of lossless speculative decoding methods on multiple video understanding tasks. The default window size

\gamma

of the baselines is set to 5, except for STD [ 52 ] with

\gamma=9

. ParallelVLM achieves adaptive draft length [ 30 ] , and we dynamically adjust the

\gamma

according to the speed ratio

c=T_{p}/T_{q}

to mitigate mutual waiting. From up to bottom,

\gamma=\left\{3,9,5,2,2\right\}

at the pruning ratio

\alpha=0.9

Methods

VideoDetailCaption

VideoMME

MVBench

MVLU

LongVideoBench

Avg.

\mathcal{A}

Speedup

\mathcal{A}

Speedup

\mathcal{A}

Speedup

\mathcal{A}

Speedup

\mathcal{A}

Speedup

\mathcal{A}

Speedup

LLaVA-OV-72B, Average Retention Ratio=10%

FastV

83.2%

1.60

\times

85.7%

1.65

\times

84.1%

1.62

\times

86.3%

1.66

\times

83.7%

1.62

\times

84.6%

1.63

\times

SparseVLM

88.5%

1.58

\times

86.2%

1.61

\times

87.9%

1.59

\times

85.8%

1.63

\times

87.1%

1.60

\times

87.1%

1.60

\times

P-Drop

86.5%

1.55

\times

88.9%

1.59

\times

89.2%

1.57

\times

87.3%

1.60

\times

87.1%

1.59

\times

87.8%

1.58

\times

DyCoke

92.1%

1.44

\times

91.5%

1.48

\times

89.7%

1.46

\times

90.3%

1.49

\times

90.9%

1.47

\times

90.9%

1.47

\times

Table 2: Comparison of lossy visual token pruning methods on multiple video understanding tasks. ParallelVLM achieves a superior speedup ratio losslessly. (The minor drops stem from the inherent uncertainty of LLMs and the precision limits of hardware numerics.)

Models	Target $T_{p}$	Draft $T_{q}$
		$\alpha=0.0$	$\alpha=0.5$	$\alpha=0.9$
LLaVA-OV (7B & 72B)	420 ( $\pm 8.64$ )	78.3 ( $\pm 3.46$ )	57.5 ( $\pm 2.38$ )	46.6 ( $\pm 1.52$ )
		$c=5$	$c=7$	$c=9$
Qwen2.5-VL (7B & 32B)	203 ( $\pm 6.89$ )	63.7 ( $\pm 2.42$ )	48.2 ( $\pm 1.66$ )	39.8 ( $\pm 1.24$ )
		$c=3$	$c=4$	$c=5$

Table 3: The speed ratio

c=T_{p}/T_{q}

with different pruning ratio

\alpha

. The forward time is reported in ms .

Figure 6: Ablation study of the pruning ratio

\alpha

. ParallelVLM strikes a trade-off between speedup and acceptance at

\alpha=0.9

Panel of Figure 7 — Figure 7: Ablation Study of UV-Prune. (a) Visualizations of selected tokens across frames. (b) Impact of UV-Prune’s verifier-guidance and similarity-variation. With the joint effort of the two components, ParallelVLM achieves robust acceptance, exceeding existing baselines.

	Sequence Length
	8K	16K	24K	32K
w/o flash.	302 ( $\pm 6.15$ )	361 ( $\pm 7.06$ )	420 ( $\pm 8.32$ )	479 ( $\pm 9.14$ )
w/ flash.	248 ( $\pm 4.42$ )	280 ( $\pm 4.67$ )	311 ( $\pm 5.21$ )	340 ( $\pm 6.08$ )
Speedup	$1.22\times$	$1.29\times$	$1.35\times$	$1.41\times$

Table 4: Integration of flash attention at different sequence lengths. The forward time is reported for LLaVA-OV-72B in ms .

Models	Target $T_{p}$	Draft $T_{q}$
		$\alpha=0.0$	$\alpha=0.5$	$\alpha=0.9$
LLaVA-OV (0.5B & 7B)	78.3 ( $\pm 3.46$ )	46.1 ( $\pm 1.21$ )	32.5 ( $\pm 0.89$ )	25.8 ( $\pm 0.32$ )
		$c=1.7$	$c=2.4$	$c=3.0$
LLaVA-OV (7B & 72B)	420 ( $\pm 8.64$ )	78.3 ( $\pm 3.46$ )	57.5 ( $\pm 2.38$ )	46.6 ( $\pm 1.52$ )
		$c=5.3$	$c=7.3$	$c=9.0$
Qwen2.5-VL (7B & 32B)	203 ( $\pm 6.89$ )	63.7 ( $\pm 2.42$ )	48.2 ( $\pm 1.66$ )	39.8 ( $\pm 1.24$ )

Table 5: Sensitivity of speed ratio

c=T_{p}/T_{q}

with the increase of pruning ratio

\alpha

. The forward time is reported in ms .

Operation	Method
Operation	Autoregressive	SD	ParallelVLM
Peak Memory	198.85	225.19	225.34
Draft Prefill	7.92	7.92	/
Target Prefill	44.23	44.23	44.23
Draft Decode	/	58.14	/
Target Decode	228.14	54.85	67.85

Table 6: Peak memory usage (GB) and latency breakdown of LLaVA-OV-7B/72B (s). The output length is set to 512.

Figure 8: An illustrative example of decoding “ The video uses ParallelVLM for speedup ” is provided to demonstrate the verification mode switching of the parallel pipeline, as well as its advancement and rollback mechanisms. Here $t$ denotes the one single forward time of the draft model and the window size $\gamma$ is set to 3.

Pruning ratio $\alpha$	0.90	0.95	0.99	1.00
Accepted Lengths $M$	6.82	6.53	5.74	4.53
Acceptance rates $\tau$	0.41	0.37	0.33	0.27
Speedup	3.43 $\times$	3.28 $\times$	2.90 $\times$	2.53 $\times$

Table 7: Experiments with LLaVA-OV-7B/72B on VideoDetailCaption under extremely higher pruning ratio (

\alpha\rightarrow 1

Figure 9: Experiments with LLaVA-OV-7B/72B on VideoDetailCaption with different draft model pruning strategies.

Figure 10: Case study for LLaVA-OV-7B/72B.

Figure 11: Case study for Qwen2.5-VL-7B/32B.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs
Apr 7, 2026Yicheng Ji, Jun Zhang, Jinpeng Chen +4Speculative Decoding LLM Inference Acceleration
ZJU · BUPT
EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMs
May 11, 2026Jiameng Li, Minye Wu, Jiezhang Cao +2Efficient VLM Inference Token Pruning
KU Leuven · Shanghai Jiaotong University · Weill Cornell Medicine
Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs
Feb 17, 2026Libo Zhang, Zhaoning Zhang, Wangyang Hong +2Speculative Decoding Long-Context VLMs
National Key Laboratory of Parallel and Distributed Computing College of Computer Science and Technology National University of Defense Technology, Changsha, China.

Apr 7, 2026J/K move · Enter open · S save

See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs

Yicheng Ji, Jun Zhang, Jinpeng Chen, Cong Wang+3

ZJU · BUPT

Speculative Decoding LLM Inference Acceleration Video-Language Models

Open paper AskPDF arXiv

Video Large Language Models (Video-LLMs) excel in video understanding but suffer from high inference latency during autoregressive generation. Speculative Decoding (SD) mitigates this by applying a draft-and-verify paradigm, yet existing methods are constrained by rigid exact-match rules, severely limiting the acceleration potential. To bridge this gap, we propose LVSpec, the first training-free loosely SD framework tailored for Video-LLMs. Grounded in the insight that generation is governed by sparse visual-relevant anchors (mandating strictness) amidst abundant visual-irrelevant fillers (permitting loose verification), LVSpec employs a lightweight visual-relevant token identification scheme to accurately pinpoint the former. To further maximize acceptance, we augment this with a position-shift tolerant mechanism that effectively salvages positionally mismatched but semantically equivalent tokens. Experiments demonstrate that LVSpec achieves high fidelity and speed: it preserves >99.8 of target performance while accelerating Qwen2.5-VL-32B by 2.70x and LLaVA-OneVision-72B by 2.94x. Notably, it boosts the mean accepted length and speedup ratio by 136% and 35% compared to SOTA training-free SD methods for Video-LLMs.

Abstract

Efficient VLM Inference

Explore similar work

See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs

EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMs

Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs