ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding

Quan Kong; Yuhao Shen; Yicheng Ji; Huan Li; Cong Wang

cs.CVMar 20, 2026

ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding

Authors: Quan Kong, Yuhao Shen, Yicheng Ji, Huan Li, Cong Wang

Organizations: Zhejiang University, Hangzhou, China

Abstract

Although current Video-LLMs achieve impressive performance in video understanding tasks, their autoregressive decoding efficiency remains constrained by the massive number of video tokens. Visual token pruning can partially ease this bottleneck, yet existing approaches still suffer from information loss and yield only modest acceleration in decoding. In this paper, we propose ParallelVLM, a training-free draft-then-verify speculative decoding framework that overcomes both mutual waiting and limited speedup-ratio problems between draft and target models in long-video settings. ParallelVLM features two parallelized stages that maximize hardware utilization and incorporate an Unbiased Verifier-Guided Pruning strategy to better align the draft and target models by eliminating the positional bias in attention-guided pruning. Extensive experiments demonstrate that ParallelVLM effectively expands the draft window by $1.6\sim1.8\times$ with high accepted lengths, and accelerates various video understanding benchmarks by 3.36 $\times$ on LLaVA-Onevision-72B and 2.42 $\times$ on Qwen2.5-VL-32B compared with vanilla autoregressive decoding.

Figures & tables

Figure 2: Comparison of different training-free speculative decoding frameworks for Video-LLMs. (a) Vanilla SD: sequential in both prefilling and decoding stages, achieving speedup only with the draft-then-verify design. (b) SpecVLM [ 17 ] : inherits vanilla SD and enables faster draft model execution via attention-guided pruning. (c) ParallelVLM: adopts a parallel pipeline to mitigate idling time in vanilla SD, co-designed with UV-Prune strategy to expand the draft window size meanwhile preserving draft/target alignment.

Figure 3: Scaling of (a) Prefilling Latency and (b) Decoding Time with increasing number of video tokens. Experiments are conducted on LLaVA-OV 7B draft model and 72B target model.

Figure 5: The proposed ParallelVLM consists of Parallel Prefilling (PP), Parallel Decoding (PD) and the Unbiased Verifier-guided Pruning (UV-Prune). (a) During PP stage: draft model prefilling and pruning executes in parallel with target model prefilling, where UV-Prune transfers salient alignment semantics from the target model to the draft model without “positional bias”. (b) During PD stage: the reduced draft decoding time $T_{q}$ enables an enlarged window size $\gamma$ in parallel with verification, meanwhile maintaining high acceptance rates $\tau$ .

Models

Methods

VideoDetailCaption

VideoMME

MVBench

MVLU

LongVideoBench

Avg.

M

Speedup

M

Speedup

M

Speedup

M

Speedup

M

Speedup

LLaVA-OV (0.5B & 7B)

Vanilla SD

4.26

1.08

\times

4.34

1.10

\times

4.18

1.05

\times

4.22

1.06

\times

4.32

1.01

\times

1.06

\times

OnSD

3.73

1.40

\times

3.67

1.38

\times

3.78

1.32

\times

3.56

1.29

\times

3.76

1.36

\times

1.35

\times

SD-Tree

5.08

1.28

\times

5.05

1.28

\times

4.97

1.25

\times

4.90

1.24

\times

5.12

1.26

\times

1.26

\times

SpecVLM

4.89

1.83

\times

4.83

1.82

\times

4.76

1.77

\times

4.71

1.76

\times

4.92

1.86

\times

1.81

\times

ParallelVLM

8.47

2.12

\times

7.86

2.06

\times

8.64

2.15

\times

8.24

2.14

\times

8.35

2.10

\times

2.11

\times

Table 1: Comparison of lossless speculative decoding methods on multiple video understanding tasks. The default window size

\gamma

of the baselines is set to 5, except for STD [ 52 ] with

\gamma=9

. ParallelVLM achieves adaptive draft length [ 30 ] , and we dynamically adjust the

\gamma

according to the speed ratio

c=T_{p}/T_{q}

to mitigate mutual waiting. From up to bottom,

\gamma=\left\{3,9,5,2,2\right\}

at the pruning ratio

\alpha=0.9

Methods

VideoDetailCaption

VideoMME

MVBench

MVLU

LongVideoBench

Avg.

\mathcal{A}

Speedup

\mathcal{A}

Speedup

\mathcal{A}

Speedup

\mathcal{A}

Speedup

\mathcal{A}

Speedup

\mathcal{A}

Speedup

LLaVA-OV-72B, Average Retention Ratio=10%

FastV

83.2%

1.60

\times

85.7%

1.65

\times

84.1%

1.62

\times

86.3%

1.66

\times

83.7%

1.62

\times

84.6%

1.63

\times

SparseVLM

88.5%

1.58

\times

86.2%

1.61

\times

87.9%

1.59

\times

85.8%

1.63

\times

87.1%

1.60

\times

87.1%

1.60

\times

P-Drop

86.5%

1.55

\times

88.9%

1.59

\times

89.2%

1.57

\times

87.3%

1.60

\times

87.1%

1.59

\times

87.8%

1.58

\times

DyCoke

92.1%

1.44

\times

91.5%

1.48

\times

89.7%

1.46

\times

90.3%

1.49

\times

90.9%

1.47

\times

90.9%

1.47

\times

Table 2: Comparison of lossy visual token pruning methods on multiple video understanding tasks. ParallelVLM achieves a superior speedup ratio losslessly. (The minor drops stem from the inherent uncertainty of LLMs and the precision limits of hardware numerics.)

Models	Target $T_{p}$	Draft $T_{q}$
		$\alpha=0.0$	$\alpha=0.5$	$\alpha=0.9$
LLaVA-OV (7B & 72B)	420 ( $\pm 8.64$ )	78.3 ( $\pm 3.46$ )	57.5 ( $\pm 2.38$ )	46.6 ( $\pm 1.52$ )
		$c=5$	$c=7$	$c=9$
Qwen2.5-VL (7B & 32B)	203 ( $\pm 6.89$ )	63.7 ( $\pm 2.42$ )	48.2 ( $\pm 1.66$ )	39.8 ( $\pm 1.24$ )
		$c=3$	$c=4$	$c=5$

Table 3: The speed ratio

c=T_{p}/T_{q}

with different pruning ratio

\alpha

. The forward time is reported in ms .

Figure 6: Ablation study of the pruning ratio

\alpha

. ParallelVLM strikes a trade-off between speedup and acceptance at

\alpha=0.9

Panel of Figure 7 — Figure 7: Ablation Study of UV-Prune. (a) Visualizations of selected tokens across frames. (b) Impact of UV-Prune’s verifier-guidance and similarity-variation. With the joint effort of the two components, ParallelVLM achieves robust acceptance, exceeding existing baselines.

	Sequence Length
	8K	16K	24K	32K
w/o flash.	302 ( $\pm 6.15$ )	361 ( $\pm 7.06$ )	420 ( $\pm 8.32$ )	479 ( $\pm 9.14$ )
w/ flash.	248 ( $\pm 4.42$ )	280 ( $\pm 4.67$ )	311 ( $\pm 5.21$ )	340 ( $\pm 6.08$ )
Speedup	$1.22\times$	$1.29\times$	$1.35\times$	$1.41\times$

Table 4: Integration of flash attention at different sequence lengths. The forward time is reported for LLaVA-OV-72B in ms .

Models	Target $T_{p}$	Draft $T_{q}$
		$\alpha=0.0$	$\alpha=0.5$	$\alpha=0.9$
LLaVA-OV (0.5B & 7B)	78.3 ( $\pm 3.46$ )	46.1 ( $\pm 1.21$ )	32.5 ( $\pm 0.89$ )	25.8 ( $\pm 0.32$ )
		$c=1.7$	$c=2.4$	$c=3.0$
LLaVA-OV (7B & 72B)	420 ( $\pm 8.64$ )	78.3 ( $\pm 3.46$ )	57.5 ( $\pm 2.38$ )	46.6 ( $\pm 1.52$ )
		$c=5.3$	$c=7.3$	$c=9.0$
Qwen2.5-VL (7B & 32B)	203 ( $\pm 6.89$ )	63.7 ( $\pm 2.42$ )	48.2 ( $\pm 1.66$ )	39.8 ( $\pm 1.24$ )

Table 5: Sensitivity of speed ratio

c=T_{p}/T_{q}

with the increase of pruning ratio

\alpha

. The forward time is reported in ms .

Operation	Method
Operation	Autoregressive	SD	ParallelVLM
Peak Memory	198.85	225.19	225.34
Draft Prefill	7.92	7.92	/
Target Prefill	44.23	44.23	44.23
Draft Decode	/	58.14	/
Target Decode	228.14	54.85	67.85

Table 6: Peak memory usage (GB) and latency breakdown of LLaVA-OV-7B/72B (s). The output length is set to 512.

Figure 8: An illustrative example of decoding “ The video uses ParallelVLM for speedup ” is provided to demonstrate the verification mode switching of the parallel pipeline, as well as its advancement and rollback mechanisms. Here $t$ denotes the one single forward time of the draft model and the window size $\gamma$ is set to 3.

Pruning ratio $\alpha$	0.90	0.95	0.99	1.00
Accepted Lengths $M$	6.82	6.53	5.74	4.53
Acceptance rates $\tau$	0.41	0.37	0.33	0.27
Speedup	3.43 $\times$	3.28 $\times$	2.90 $\times$	2.53 $\times$

Table 7: Experiments with LLaVA-OV-7B/72B on VideoDetailCaption under extremely higher pruning ratio (

\alpha\rightarrow 1

Figure 9: Experiments with LLaVA-OV-7B/72B on VideoDetailCaption with different draft model pruning strategies.

Figure 10: Case study for LLaVA-OV-7B/72B.

Figure 11: Case study for Qwen2.5-VL-7B/32B.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

Apr 7, 2026

See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs

Video Large Language Models (Video-LLMs) excel in video understanding but suffer from high inference latency during autoregressive generation. Speculative Decoding (SD) mitigates this by applying a draft-and-verify paradigm, yet existing methods are constrained by rigid exact-match rules, severely limiting the acceleration potential. To bridge this gap, we propose LVSpec, the first training-free loosely SD framework tailored for Video-LLMs. Grounded in the insight that generation is governed by sparse visual-relevant anchors (mandating strictness) amidst abundant visual-irrelevant fillers (permitting loose verification), LVSpec employs a lightweight visual-relevant token identification scheme to accurately pinpoint the former. To further maximize acceptance, we augment this with a position-shift tolerant mechanism that effectively salvages positionally mismatched but semantically equivalent tokens. Experiments demonstrate that LVSpec achieves high fidelity and speed: it preserves >99.8 of target performance while accelerating Qwen2.5-VL-32B by 2.70x and LLaVA-OneVision-72B by 2.94x. Notably, it boosts the mean accepted length and speedup ratio by 136% and 35% compared to SOTA training-free SD methods for Video-LLMs.

Yicheng Ji, Jun Zhang, Jinpeng Chen +4

ZJU · BUPT

Speculative Decoding LLM Inference Acceleration Video-Language Models

Ask PDF

May 11, 2026

EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMs

Long-form video understanding remains challenging for Video Large Language Models (VideoLLMs), as the dense frame sampling introduces massive visual tokens while sparse sampling risks missing critical temporal evidence and leading to LLM hallucination. Existing training-free token reduction methods either treat videos equally as static images or rely on segment-level merging heuristics, which weaken fine-grained spatiotemporal modeling and introduce additional overhead. In this paper, we propose EchoPrune, a lightweight and training-free token pruning method that improves temporal resolution under a fixed LLM-side visual token budget. Our core idea is to interpret redundant video tokens as temporal echoes: if a token is well reconstructed from the previous frame, it is merely a temporally redundant echo; otherwise, it may capture new events, motion, or query-relevant visual evidence. Based on this insight, EchoPrune scores visual tokens by (i) query-guided crossmodal relevance and (ii) temporal reconstruction error, measured by correspondence matching and echo matching across consecutive frames. The selected tokens preserve task-relevant cues and temporal novelty while suppressing predictable redundancy, allowing VideoLLMs to observe more frames without increasing the decoding budget. Extensive experiments on LLaVA-OV, Qwen2.5VL, and Qwen3VL across six video understanding benchmarks show that EchoPrune enables VideoLLMs to process up to 20x frames under the same token budget, yielding improved performance (+8.6%) and inference speedup (5.6x for prefilling) on Qwen2.5VL-7B.

Jiameng Li, Minye Wu, Jiezhang Cao +2

KU Leuven · Shanghai Jiaotong University · Weill Cornell Medicine

Efficient VLM Inference Token Pruning Long-Video Understanding Video Token Compression Visual Token Pruning

Ask PDF

Feb 17, 2026

Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs

Although speculative decoding is widely used to accelerate Vision-Language Models (VLMs) inference, it faces severe performance collapse when applied to Video Large Language Models (Vid-LLMs). The draft model typically falls into the trap of attention dilution and negative visual gain due to key-value cache explosion and context window mismatches. We observe a visual semantic internalization phenomenon in Vid-LLMs, indicating that critical visual semantics are implicitly encoded into text hidden states during deep-layer interactions, which renders raw visual inputs structurally redundant during deep inference. To address this, we propose the Sparrow framework, which first utilizes visually-aware text-anchored window attention via hidden state reuse to fully offload visual computation to the target model, and leverages intermediate-layer visual state bridging to train the draft model with semantic-rich intermediate states, thereby filtering out low-level visual noise. Additionally, a multi-token prediction strategy is introduced to bridge the training-inference distribution shift. Experiments show that Sparrow achieves an average speedup of 2.82x even with 25k visual tokens, effectively resolving the performance degradation in long sequences and offering a practical solution for real-time long video tasks.

Libo Zhang, Zhaoning Zhang, Wangyang Hong +2

National Key Laboratory of Parallel and Distributed Computing College of Computer Science and Technology National University of Defense Technology, Changsha, China.

Speculative Decoding Long-Context VLMs

Ask PDF

Abstract

Efficient VLM Inference

Explore similar work

See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs

EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMs

Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs