V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models

Xinying Lin; Xuyang Liu; Yiyu Wang; Teng Ma; Wenqi Ren

cs.CVMar 29, 2026

V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models

Authors: Xinying Lin, Xuyang Liu, Yiyu Wang, Teng Ma, Wenqi Ren

Organizations: Shenzhen Campus of Sun Yat-sen University · Shenzhen Loop Area Institute · Sichuan University · EPIC Lab, Shanghai Jiao Tong University

Abstract

Video large language models (VideoLLMs) show strong capability in video understanding, yet long-context inference is still dominated by massive redundant visual tokens in the prefill stage. We revisit token compression for VideoLLMs under a tight budget and identify a key bottleneck, namely insufficient spatio-temporal information coverage. Existing methods often introduce discontinuous coverage through coarse per-frame allocation or scene segmentation, and token merging can further misalign spatio-temporal coordinates under MRoPE-style discrete (t,h,w) bindings. To address these issues, we propose V-CAST (Video Curvature-Aware Spatio-Temporal Pruning), a training-free, plug-and-play pruning policy for long-context video inference. V-CAST casts token compression as a trajectory approximation problem and introduces a curvature-guided temporal allocation module that routes per-frame token budgets to semantic turns and event boundaries. It further adopts a dual-anchor spatial selection mechanism that preserves high-entropy visual evidence without attention intervention, while keeping retained tokens at their original coordinates to maintain positional alignment. Extensive experiments across multiple VideoLLMs of different architectures and scales demonstrate that V-CAST achieves 98.6% of the original performance, outperforms the second-best method by +1.1% on average, and reduces peak memory and total latency to 86.7% and 86.4% of vanilla Qwen3-VL-8B-Instruct.

Figures & tables

Figure 2: Spatio-temporal budgeting profiles. We visualize per-frame token budgets (higher means more retained tokens) on representative short, medium, and long videos. Uniform Allocation Yang et al. (2025a) stays nearly flat and misses brief evidence peaks; segment-based pinelines Shen et al. (2025) ; Shao et al. (2025a) introduce boundary-sensitive budget jumps and waste tokens within redundant windows; global-uniqueness budgeting Liu et al. (2025c) can over-concentrate on a few globally distinctive frames and under-cover transitional segments. Curvature-aware budgeting allocates more tokens to rapid semantic changes and event boundaries, improving spatiotemporal information coverage under tight budgets.

Figure 4: Overview of V-CAST. V-CAST formulates token pruning for VideoLLMs as an optimal semantic-trajectory approximation problem under a fixed budget. It applies Curvature-Guided Temporal Allocation to assign per-frame budgets by tracking semantic transitions, and then performs Dual-Anchor Spatial Token Selection to retain diverse and salient tokens within each frame while preserving their original on-grid coordinates.

Methods	MVBench	LongVideo Bench	MLVU	VideoMME				Average
Methods	MVBench	LongVideo Bench	MLVU	Overall	Short	Medium	Long	Score	%
Max Input Frames=64
Qwen3-VL-8B-Instruct	69.2	62.8	68.9	66.9	78.8	66.2	55.8	67.0	100.0
Retention Ratio=25%
VisionZip (CVPR’25)	64.8	58.9	63.4	63.1	73.7	60.3	55.4	62.6	93.4
VidCom 2 (EMNLP’25)	67.5	59.6	64.0	64.9	75.4	63.4	55.9	64.0	95.5

Table 2: Performance comparison with other baselines on Qwen3-VL-8B-Instruct across benchmarks under multiple settings. “Average” shows the mean performance across benchmarks.

Methods	MVBench	LongVideo Bench	MLVU	VideoMME				Average
Methods	MVBench	LongVideo Bench	MLVU	Overall	Short	Medium	Long	Score	%
LLaVA-OV-7B	58.3	56.6	63.1	58.4	69.9	56.7	48.8	59.1	100.0
Retention Ratio=25%
FastV (ECCV’24)	55.5	53.3	59.6	55.3	65.0	53.8	47.0	55.9	94.6
PDrop (CVPR’25)	55.3	51.3	57.1	55.5	64.7	53.1	48.7	54.8	92.7
SparseVLM (ICML’25)	56.4	53.9	60.7	57.3	68.4	55.2	48.1	57.1	96.6

Table 3: Performance comparison with other baselines with LLaVA-OV-7B across different benchmarks. We use the default 32-frame input setting.

Methods	MVBench	LongVideo Bench	MLVU	VideoMME				Average
Methods	MVBench	LongVideo Bench	MLVU	Overall	Short	Medium	Long	Score	%
LLaVA-Video-7B	60.4	58.9	67.3	64.4	77.3	62.4	53.4	62.8	100.0
Retention Ratio=25%
FastV (ECCV’24)	52.1	54.8	57.8	58.6	68.7	58.4	48.7	55.8	88.9
SparseVLM (ICML’25)	55.4	54.2	58.9	60.1	71.1	59.1	50.1	57.2	91.1
VisionZip (CVPR’25)	57.9	56.3	62.6	62.5	73.6	62.3	51.9	59.8	95.2

Table 4: Performance comparison with other baselines with LLaVA-Video-7B across different benchmarks. We use the default 64-frame setting.

Methods	MVBench	LongVideo Bench	MLVU	VideoMME				Average
Methods	MVBench	LongVideo Bench	MLVU	Overall	Short	Medium	Long	Score	%
Qwen3-VL-32B-Instruct	73.2	62.4	66.1	69.3	78.3	67.1	62.4	67.8	100.0
Retention Ratio=25%
VisionZip (CVPR’25)	62.9	60.0	62.2	64.8	72.8	63.1	58.6	62.5	92.2
VidCom 2 (EMNLP’25)	70.2	60.4	64.9	67.0	75.6	64.3	61.2	65.6	96.8
FastVID (NeurIPS’25)	71.0	60.9	64.5	65.1	75.1	62.6	57.7	65.4	96.5

Table 5: Performance comparison with other baselines on the larger model Qwen3-VL-32B-Instruct across different benchmarks. We use 32-frame input setting.

Methods	MVBench	LongVideo Bench	MLVU	VideoMME				Average Score
Methods	MVBench	LongVideo Bench	MLVU	Overall	Short	Medium	Long	Average Score
Qwen3-VL-30B-A3B-Instruct	OOM	OOM	OOM	OOM	OOM	OOM	OOM	-
Retention Ratio=25%
VisionZip (CVPR’25)	61.3	62.4	64.1	62.7	70.7	61.3	56.0	62.6
VidCom 2 (EMNLP’25)	69.7	65.5	68.5	68.1	79.1	68.0	58.9	68.0
FastVID (NeurIPS’25)	68.2	60.7	64.8	62.4	70.3	61.6	55.2	64.0

Table 6: Performance comparison with other baselines on the MoE-based model Qwen3-VL-30B-A3B-Instruct across different benchmarks. We use 64-frame input setting.

Figure 5: Consistent gains with more frames. Performance trends on LongVideoBench, MLVU, VideoMME (Long), and EgoSchema as input frames increase. V-CAST improves accuracy and scales to longer inputs, while some baselines show limited gains or OOM failures at larger frame counts.

Methods	Prefilling $\downarrow$	LLM Generation $\downarrow$	Total Latency $\downarrow$	GPU Peak $\downarrow$	Throughput $\uparrow$	Performance $\uparrow$
	Latency (s)	Latency (s)	(s)	Memory (MB)	(item/s)
Qwen3-VL-8B-Instruct	243.6	280.9	1440.4	22478.0	1.87	64.5
Random	116.9 (↓52.0%)	147.3 (↓47.6%)	1271.9 (↓11.7%)	19547.5 (↓13.0%)	2.12 (1.13 $\times$ )	61.5 (↓3.0)
VisionZip (CVPR’25)	155.6 (↓36.1%)	190.5 (↓32.2%)	1276.8 (↓11.4%)	19974.7 (↓11.2%)	2.11 (1.13 $\times$ )	60.1 (↓4.4)
VidCom 2 (EMNLP’25)	119.0 (↓51.1%)	150.2 (↓46.5%)	1271.0 (↓11.8%)	19547.5 (↓13.0%)	2.12 (1.13 $\times$ )	62.4 (↓2.1)
FastVID (NeurIPS’25)	120.6 (↓50.5%)	153.6 (↓45.3%)	1317.6 (↓8.5%)	19547.5 (↓13.0%)	2.05 (1.10 $\times$ )	60.5 (↓4.0)

Table 7: Efficiency comparisons. Qwen3-VL-8B-Instruct on VideoMME at

R=25\%

(max input frames = 32). “Prefill Latency”: prompt-to-first-token; “LLM Generation Latency”: decoding time (first-to-last-token); “Total Latency”: end-to-end wall-clock time in our experimental setup; “Throughput”: samples per second.

Figure 6: Efficiency comparison on LLaVA-OneVision-7B. We compare Vanilla, FastVID, VidCom 2 , and V-CAST on Prefill Latency, Total Latency, and Peak Memory.

Table 8: Ablation on Temporal Allocation. Evaluating temporal routing metrics on Qwen3-VL-8B at

R=25\%

under different maximum input frames.

Spatial Selection	MVBench	MLVU	VideoMME			Average
Spatial Selection			Overall	Short	Medium	Score
Qwen3-VL-8B-Instruct	68.6	63.5	64.5	76.0	60.4	65.5
Contextual Diversity Only	67.5	61.6	61.2	71.3	57.1	63.4
Feature Activation Only	63.0	58.6	61.4	70.0	57.2	61.0
Dual-Anchor (Ours)	67.9	62.7	63.5	74.4	60.2	64.7

Table 9: Ablation on Spatial Selection. Evaluating spatial selection anchors on Qwen3-VL-8B at

R=25\%

Figure 7: Visualization of frame budget allocation. We compare Uniform allocation , Global Uniqueness (VidCom 2 Liu et al. (2025c) ), and Curvature-Aware allocation (V-CAST) under a $R=25\%$ . A higher curve indicates a larger per-frame token budget.

Figure 8: Qualitative comparison. V-CAST highlights task-critical moments and yields correct answers where baselines and even the vanilla model fail.

Figure 9: Broad applicability of V-CAST budgeting on other frame compression methods. “+V-CAST” denotes equipping the original method with our curvature-aware budgeting strategy.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	MVBench	LongVideoBench	MLVU	VideoMME				Average
Methods	MVBench	LongVideoBench	MLVU	Overall	Short	Medium	Long	Score	%
Max Input Frames=64
Qwen3-VL-8B-Instruct	69.2	62.8	68.9	66.9	78.8	66.2	55.8	67.0	100.0
Retention Ratio=35%
VisionZip (CVPR’25)	65.7	58.9	64.3	64.3	74.9	62.2	55.9	63.3	94.5
VidCom 2 (EMNLP’25)	68.9	61.0	65.5	66.0	77.3	64.9	55.8	65.4	97.6

Appendix

Table 10: Performance comparison with other baselines on Qwen3-VL-8B-Instruct under a retention ratio of

35\%

. “Average” denotes the mean performance across benchmarks.

Table 11: Sensitivity analysis on V-CAST hyper-parameters. We evaluate (a) temporal temperature

\tau

and (b) spatial selection weights on Qwen3-VL-8B-Instruct at

R=25\%

Figure 10: Additional qualitative comparisons. V-CAST preserves task-critical moments and correctly answers challenging cases where baseline compression methods, and occasionally even the vanilla model, miss decisive visual evidence.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

GSTEP: Global Spatio-Temporal Density-Driven Visual Token Pruning for Efficient Video Large Language Models
Aug 4, 2026Mengjie Zhang, Qihui Zhu, Tao Zhang +10Efficient VLM Inference Video-Language Models
University of Science and Technology of China He Fei, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center He Fei, China
Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention
Mar 23, 2026Junhao Du, Jialong Xue, Anqi Li +2Video Token Compression Large Vision-Language Models
Shanghai Jiao Tong University · Beijing University of Posts and Telecommunications
EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMs
May 11, 2026Jiameng Li, Minye Wu, Jiezhang Cao +2Efficient VLM Inference Token Pruning
KU Leuven · Shanghai Jiaotong University · Weill Cornell Medicine

Aug 4, 2026J/K move · Enter open · S save

GSTEP: Global Spatio-Temporal Density-Driven Visual Token Pruning for Efficient Video Large Language Models

Mengjie Zhang, Qihui Zhu, Tao Zhang, Shuangwu Chen+9

University of Science and Technology of China He Fei, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center He Fei, China

Efficient VLM Inference Video-Language Models Visual Token Pruning

Open paper AskPDF arXiv

Video large language models (VideoLLMs) achieve strong video understanding performance, but their inference remains expensive due to the large number of redundant spatio-temporal visual tokens in long videos. Existing token pruning methods alleviate this cost by reducing redundant tokens, yet most of them rely on segment-level local pruning, where videos are partitioned into isolated segments and tokens are selected independently within each segment. Such designs may under-preserve short but semantically dense segments and discard tokens that appear non-salient locally but remain critical from a global perspective. To address this issue, we propose GSTEP (Global Spatio-Temporal Density Pruning), a plug-and-play pruning framework that models video as a continuous spatio-temporal information flow. GSTEP constructs a token-level spatio-temporal density by combining a continuous temporal density, obtained from a smoothed centered frame-level change signal, with intra-frame spatial density, and then performs global token sampling by jointly balancing information density and coverage. Extensive experiments on multiple VideoLLMs and public benchmarks demonstrate that GSTEP consistently achieves strong accuracy-efficiency trade-offs and generalizes well across model architectures and evaluation settings. On LLaVA-OneVision-7B, GSTEP prunes 75% of visual tokens, preserves up to 100.2% of the original average performance across benchmarks, and achieves a 1.17 end-to-end speedup.

Abstract

Efficient VLM Inference

Explore similar work

GSTEP: Global Spatio-Temporal Density-Driven Visual Token Pruning for Efficient Video Large Language Models

Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention

EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMs