Training-Free Inference Acceleration for Video Large Language Models: Methods, Trade-offs, and Evaluation

Yuqi Li; Chuanyue Yan; Neil Rodrigues; Xianglong Wang; Xiaoqin Feng; Junming Liu; Xi Xiao; Zhongwei Cheng

cs.CVOct 26, 2026

Training-Free Inference Acceleration for Video Large Language Models: Methods, Trade-offs, and Evaluation

Authors: Yuqi Li, Chuanyue Yan, Neil Rodrigues, Xianglong Wang, Xiaoqin Feng, Junming Liu, Xi Xiao, Zhongwei Cheng,

Organizations: Wyze Labs, Inc. · The City University of New York · Wuhan Institute of Technology · University of Melbourne ·

Abstract

Video Large Language Models (VLMs) have demonstrated strong capabilities in video understanding and reasoning, but processing long visual sequences incurs substantial computational, latency, and memory overhead. This survey examines training-free inference acceleration, focusing on reducing inference costs while preserving task-relevant spatiotemporal evidence. We organize existing methods by their intervention stages in the inference pipeline, covering input, visual encoding, projector, prefill, and decoding. Post-training quantization (PTQ) is discussed separately as a cross-stage extension with varying calibration requirements. We analyze how intervention position affects evidence preservation and computational savings, and why local efficiency improvements may not translate into end-to-end acceleration. We complement this analysis with quantitative evidence from prior studies and our own experiments under a shared evaluation setting to assess accuracy-latency trade-offs.

Figures & tables

Figure 1: General architecture and inference pipeline of a VLM, comprising a vision encoder, a projector connecting visual and language representations, and an LLM with prefill and decoding stages.

Figure 2: Stage-wise taxonomy of training-free VLM inference acceleration with representative methods. PTQ is included as a related cross-stage extension.

Method	Stage	Video-MME		EgoSchema		LongVideoBench		MVBench		QLB ${}_{\mathrm{TTFT}}\uparrow$
Method	Stage	Acc. $\uparrow$	TTFT $\downarrow$	Acc. $\uparrow$	TTFT $\downarrow$	Acc. $\uparrow$	TTFT $\downarrow$	Acc. $\uparrow$	TTFT $\downarrow$	QLB ${}_{\mathrm{TTFT}}\uparrow$
Full input	–	58.48	294	64.20	298	55.20	301	58.39	294	100.0
Curv. merge	Proj.	57.89	236	64.60	239	54.76	240	58.03	236	110.6
PriorTR	Pref.	57.70	294	63.00	296	54.27	291	57.89	255	100.7
EarlyTom	Enc.+Proj.	57.85	160	62.80	163	54.00	160	56.08	161	126.6
FrameFusion	Pref.	57.85	342	63.40	386	54.93	337	58.39	275	93.3

Table 1: Quality–latency comparison. Stage denotes the compression location: Enc. (Encoding), Proj. (Projector), and Pref. (Prefill). Accuracy is in % and TTFT in ms. Higher accuracy and QLB are better; lower TTFT is better.

Method	Ret.	Quality $\uparrow$	TPOT $\downarrow$	QLB ${}_{\mathrm{TPOT}}\uparrow$
EgoSchema: Qwen2.5-VL-3B, Acc. (%)
Full KV	100%	57.20	59.37	100.0
VisCache	40%	52.00	56.82	92.9
VisCache	28%	55.00	49.50	104.9
VisCache	19%	54.60	47.70	105.9
VideoDC: LLaVA-OneVision-7B, VDCscore

Table 2: Quality–latency trade-offs during decoding. Reported hardware includes 4

\times

A100 (80 GB) for VisCache and A6000 (48 GB) for DyCoke. Ret. denotes KV-cache or visual-token retention (%); Quality reports EgoSchema accuracy (%) or VideoDC captioning score, and TPOT is in ms/token.

LLaVA-OneVision-7B, 8 benchmarks
Method	Bits	Avg. (Q/FP16)	Q-ret. $\uparrow$
MBQ	W3A16	64.4/71.4	90.2%
VLMQ	W3A16	65.7/71.4	92.0%
VLM-PTQ	W3A16	69.5/71.4	97.3%
LLaVA-OneVision-7B, 5 benchmarks
MBQ	W3A16	70.44/73.37	96.0%

Table 3: PTQ quality retention under the reported model, bit-width, and benchmark settings. Q/FP16 denotes the average scores of quantized and FP16 models; Q-ret. denotes their ratio (%).

Method	Acc. $\uparrow$	Prefill $\downarrow$	Gen. $\downarrow$	E2E $\downarrow$	Peak Mem. $\downarrow$
Qwen3-VL-8B	64.5	243.6	280.9	1440.4	22478.0
VisionZip	60.1	155.6	190.5	1276.8	19974.7
VidCom 2	62.4	119.0	150.2	1271.0	19547.5
HoliTom	59.7	134.3	160.8	1261.8	19974.7
FlashVID	62.3	146.6	175.0	1308.7	41363.6
V-CAST	63.5	118.8	149.9	1245.1	19547.5

Table 4: System-level quality and efficiency on Video-MME using Qwen3-VL-8B-Instruct with up to 32 frames and 25% token retention for compressed methods. Experiments use NVIDIA H100 (80 GB) GPUs. Prefill and Gen. denote the reported pre-generation and autoregressive generation times, respectively. Accuracy is in %, aggregate latency in seconds, and peak GPU memory in MB.

Metric	Formulation	Scope	Definition
Task Quality and Compression
Accuracy	$\displaystyle\frac{N_{\mathrm{correct}}}{N_{\mathrm{total}}}\times 100\%$	Quality	Percentage of correctly answered questions under the corresponding benchmark protocol.
VDCscore	Benchmark-reported score	Quality	Video captioning quality measured using the VideoDC evaluation protocol.
Ret.	$\displaystyle\frac{N_{\mathrm{retained}}}{N_{\mathrm{original}}}\times 100\%$	Compression	Percentage of retained visual tokens or KV states relative to the source-defined budget.
Q-ret.	$\displaystyle\frac{Q_{\mathrm{quant}}}{Q_{\mathrm{FP16}}}\times 100\%$	PTQ	Quantized-model quality relative to the matched FP16 baseline.
Bits	$\mathrm{W}b_{w}\mathrm{A}b_{a}$	PTQ	Numerical precision of weights and activations; KV-cache precision is specified separately when applicable.

Table 5: Core metrics for evaluating task quality, compression, and realized inference efficiency. QLB measures the relative quality–latency balance under matched evaluation settings.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmarks used in quantitative comparisons
Benchmark	Venue	Year	Video Length	Scale	Primary Focus
Video-MME	CVPR	2025	11 s–1 h	900 videos, 2.7K QA	General video understanding across durations
EgoSchema	NeurIPS	2023	Avg. 180 s	5.0K videos, 5.0K QA	Long-horizon egocentric reasoning
LongVideoBench	NeurIPS	2024	Up to 1 h	3.8K videos, 6.7K QA	Long-context referring and temporal reasoning
MVBench	CVPR	2024	Avg. 16 s	3.6K videos, 4.0K QA	Multi-task temporal understanding
VideoDC (VDC)	ICLR	2025	–	$\sim$ 1.0K clips, captions	Detailed video captioning and VDCscore

Appendix

Table 6: Representative benchmarks for video understanding. The first group includes datasets used in the main-text quantitative comparisons; the second provides broader task coverage. Dataset sizes are approximate and may vary across evaluation subsets.

Input Stage
Method	Host VLM	Ret. / Budget	NExT	MVB	V-MME	ES	LVB	MLVU
AKS	LLaVA-Video-7B	64f	–	–	65.3	–	62.7	–
BOLT	LLaVA-OneVision-7B	1fps $\rightarrow$ 8f	77.4	–	56.1	59.23	54.5	63.6
MDP 3	LLaVA-OneVision-7B	128 $\rightarrow$ 8f	–	–	59.6	–	59.0	69.8
VideoTree	GPT-4	Adaptive	75.6	–	–	61.1	–	–
Q-Frame	Qwen2-VL-7B	4H+8M+32L	–	–	58.3	–	58.4	65.4

Appendix

Table 7: Reported performance of representative VLM acceleration methods, grouped by their primary intervention stage. NExT, MVB, V-MME, ES, LVB, and MLVU denote NExT-QA, MVBench, Video-MME, EgoSchema, LongVideoBench, and MLVU, respectively. Ret./Budget follows the source-defined input, token, cache, or KV budget and is not directly comparable across methods. “–” denotes an unreported result; † identifies a stage-specific component or configuration of a multi-stage method; C denotes a cross-stage method. Results from different hosts or protocols should not be interpreted as direct rankings.

Backbone / Setting	Offline Video Understanding			Streaming Video Understanding
	MVBench $\uparrow$	EgoSchema $\uparrow$	V-MME Avg. $\uparrow$	SBench $\uparrow$	OVO Avg. $\uparrow$
LLaVA-OneVision-7B
Base	57.02	59.93	57.67	71.34	53.35
+ HERMES (4K)	56.92	60.29	58.85	73.23	58.27
Qwen2.5-VL-7B
Base	65.00	58.47	64.52	73.31	52.28

Appendix

Table 8: Offline and streaming video understanding performance reported by HERMES (%). SBench denotes the real-time visual understanding evaluation of StreamingBench. OVO Avg. averages the real-time visual perception and backward tracing categories of OVO-Bench. HERMES (4K) specifies a 4K-token KV-cache budget. Offline and streaming benchmarks use different evaluation protocols; scores are compared only within the same benchmark.

Model	METEOR			GPT4-Score
	Single $\uparrow$	Multi $\uparrow$	$\Delta_{\mathrm{ctx}}$	Single $\uparrow$	Multi $\uparrow$	$\Delta_{\mathrm{ctx}}$
InternVL2	26.85	32.74	+5.89	32.26	42.25	+9.99
VILA	28.44	33.57	+5.13	34.63	45.41	+10.78
InternLM-XComposer2.5	23.32	31.02	+7.70	37.11	49.69	+12.58
MiniCPM-V 2.6	27.36	33.90	+6.54	36.73	48.57	+11.84
GPT-4V	28.66	36.47	+7.81	44.94	60.11	+15.17

Appendix

Table 9: Single-instance (Single) and multi-turn (Multi) video QA performance on SVBench. METEOR measures reference-based text similarity, including synonym and stemming matches; GPT4-Score evaluates answer correctness using GPT-4 as a judge. Both metrics are shown on the reported 0–100 score scale.

\Delta_{\mathrm{ctx}}

is the derived difference between Multi and Single, in score points. It does not measure inference acceleration or isolate the causal effect of dialogue history.

Target / Draft Setting	Method	$M\uparrow$	Speedup $\uparrow$	$\Delta M\uparrow$	$\Delta S\uparrow$	Fidelity
Five-benchmark matched evaluation in ParallelVLM
LLaVA-OV 72B / 7B	SpecVLM	4.42	2.74 $\times$	–	–	Exact
LLaVA-OV 72B / 7B	ParallelVLM	6.83	3.36 $\times$	+54.4%	+22.6%	Exact
Qwen2.5-VL 32B / 7B	SpecVLM	4.19	2.11 $\times$	–	–	Exact
Qwen2.5-VL 32B / 7B	ParallelVLM	4.28	2.42 $\times$	+2.2%	+14.7%	Exact
Video Detail Caption evaluation in LVSpec

Appendix

Table 10: Matched-protocol speculative decoding results. Target/Draft identifies the verifier and drafter model sizes, respectively.

M

is the mean number of tokens accepted per speculative iteration. Speedup is the source-reported decoding acceleration relative to autoregressive generation (

1.00\times

\Delta M

and

\Delta S

are percentage improvements over SpecVLM within each matched block; the five-benchmark

\Delta M

values are calculated from unrounded averages. The first block covers Video Detail Caption, Video-MME, MVBench, MLVU, and LongVideoBench; the second uses Video Detail Caption (VDC). “Exact” indicates strict target-distribution preservation, whereas percentages indicate task-performance retention under relaxed verification, not exact output agreement. Results are comparable only within matched blocks.

Efficient Inference

Efficient inference reduces the compute, memory, latency, or energy needed to produce accurate predictions, posterior estimates, and decisions.

See all 109 papers

LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs
May 17, 2026Jihwan Kim, Nikhil Parthasarathy, Danfeng Qin +5Efficient VLM Inference Video-Language Models
1Google DeepMind · 2Seoul National University
VETO: Video Efficient Token Optimization for Vision Language Models
Oct 1, 2026Gueter Josmy Faure, Hao Ping Wang, Min-Hung Chen +1Efficient VLM Inference Large Vision-Language Models
National Taiwan University · NVIDIA
Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
Mar 30, 2026Surendra Pathak, Bo HanEfficient VLM Inference Visual Token Compression
Computer Science Department, George Mason University, Fairfax, VA - 22030, USA

May 17, 2026J/K move · Enter open · S save

LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs

Jihwan Kim, Nikhil Parthasarathy, Danfeng Qin, Junhwa Hur+4

1Google DeepMind · 2Seoul National University

Efficient VLM Inference Video-Language Models Long-Video Understanding Knowledge Distillation Visual Token Compression

Open paper AskPDF arXiv

The fundamental challenge in scaling Video Large Language Models (Video LLMs) to long-form video lies in managing the explosion of visual-token context length. Existing strategies predominantly focus on "post-hoc" token reduction -- reducing visual tokens after feature extraction to alleviate the LLM's computational overhead. While these methods effectively reduce the number of visual tokens, we observe that the primary latency bottleneck then shifts from the LLM to the expensive per-frame processing of the vision encoder. To address this, we introduce LiteFrame, a strong, yet highly efficient video encoder backbone for Video LLMs. To train LiteFrame, we propose Compressed Token Distillation (CTD), a novel training framework that teaches a compact student vision encoder to directly predict information-dense, spatio-temporally compressed representations produced by a large teacher vision model, effectively bypassing redundant computation. When coupled with further Language Model Adaptation (LMA), this approach results in a new latency-accuracy Pareto frontier -- compared with InternVL3-8B, LiteFrame provides a 35% reduction in end-to-end latency while processing 8

\times

more frames and improves average video understanding accuracy across multiple benchmarks. Our results demonstrate a new potential path to unlocking longer-form video understanding under fixed compute budgets.

Abstract

Efficient Inference

Explore similar work

LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs

VETO: Video Efficient Token Optimization for Vision Language Models

Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies