ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modeling

Shaobo Ju; Baiyang Song; Tao Chen; Jiapeng Zhang; Qiong Wu; Chao Chang; HuaiXi Wang; Yiyi Zhou

cs.CVMar 24, 2026

ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modeling

Authors: Shaobo Ju, Baiyang Song, Tao Chen, Jiapeng Zhang, Qiong Wu, Chao Chang, HuaiXi Wang, Yiyi Zhou, +1 more

Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · National University of Defense Technology.

Abstract

Due to the great saving of computation and memory overhead, token compression has become a research hot-spot for MLLMs and achieved remarkable progress in image-language tasks. However, for the video, existing methods still fall short of high-ratio token compression. We attribute this shortcoming to the insufficient modeling of temporal and continual video content, and propose a novel and training-free token pruning method for video MLLMs, termed ForestPrune, which achieves effective and high-ratio pruning via Spatial-temporal Forest Modeling. In practice, ForestPrune construct token forests across video frames based on the semantic, spatial and temporal constraints, making an overall comprehension of videos. Afterwards, ForestPrune evaluates the importance of token trees and nodes based on tree depth and node roles, thereby obtaining a globally optimal pruning decision. To validate ForestPrune, we apply it to two representative video MLLMs, namely LLaVA-Video and LLaVA-OneVision, and conduct extensive experiments on a bunch of video benchmarks. The experimental results not only show the great effectiveness for video MLLMs, e.g., retaining 95.8% average accuracy while reducing 90% tokens for LLaVA-OneVision, but also show its superior performance and efficiency than the compared token compression methods, e.g., +10.1% accuracy on MLVU and -81.4% pruning time than FrameFusion on LLaVA-Video.

Figures & tables

Figure 1: Comparison between the proposed ForestPrune (Ours) and the existing compression methods [ 20 , 6 , 60 , 13 ] on VideoMME. The base model used is LLaVA-Video-7B. As pruning ratio increases, existing methods will encounter obvious performance drops, while ForestPrune is still robust.

Figure 2: Visualization of token pruning by G-Prune [ 20 ] and our ForestPrune. The image-centric G-Prune can well keep the important tokens for each frame, but also leads to obvious redundancy across frames. In contrast, our ForestPrune can obtain a globally optimal pruning via spatial-temporal forest modeling.

Figure 3: Illustration of the proposed ForestPrune. Input video frames are first encoded by the visual encoder, based on which ForestPrune will select a set of tokens of each frame as the candidate nodes. Afterwards, ForestPrune constructs the token trees based on the semantic similarity $\tau_{s}$ , spatial distance $\tau_{p}$ and the frame temporal orders, thereby forming the spatial-temporal forest (a). When obtaining excessive trees (root nodes), we will merge them before pruning (b). Then, we sort the trees in a descending order of depth (c) and then progressively prune the leaf and tail nodes until meeting the compression budget (d). Via the spatial-temporal modeling, ForestPrune can well estimate the frame-wise redundancy and obtain a globally optimal pruning decision.

Compression Ratio	Method	NExT-QA		VideoMME		MLVU		LongVideoBench		MVBench		Avg
Compression Ratio	Method	Acc $\uparrow$	Retain $\uparrow$	Acc $\uparrow$	Retain $\uparrow$	Acc $\uparrow$	Retain $\uparrow$	Acc $\uparrow$	Retain $\uparrow$	Acc $\uparrow$	Retain $\uparrow$	Retain $\uparrow$
-	LLaVA-Video 7B	82.9	100.0%	62.8	100.0%	71.3	100.0%	60.4	100.0%	61.2	100.0%	-
70%	FastV [ 6 ]	81.3	98.1%	60.0	95.5%	64.2	90.0%	56.8	94.0%	57.5	94.0%	94.5%
	VisionZip [ 60 ]	82.3	99.3%	61.3	97.6%	69.8	97.9%	58.8	97.4%	57.7	94.3%	97.4%
	G-Prune [ 20 ]	82.2	99.2%	61.7	98.2%	67.2	94.2%	57.8	95.7%	58.6	95.8%	96.7%
	STTM [ 18 ]	81.2	98.0%	62.0	98.7%	68.8	96.5%	57.7	95.5%	-	-	97.2%

Table 1: Comparison between ForestPrune and existing compression methods on LLaVA-Video-7B and LLaVA-OV 7B. Compression Ratio denotes the target reduction of visual tokens. Retain reflects the degree to which MLLM retains performance.

Model	Size	Frames	Input tokens	VideoMME	MLVU
Model	Size	Frames	Input tokens	Acc $\uparrow$	Acc $\uparrow$
GPT-4V/4T [ 1 ]	-	10	-	59.9	49.2
Qwen2.5-VL [ 3 ]	7B	64	$>12544$	65.1	69.6
Qwen3-VL [ 47 ]	8B	2fps	$>12544$	71.4	78.1
GLM-4.1V [ 14 ]	9B	64	16384	68.2	71.5
InternVL3 [ 70 ]	8B	64	16384	66.3	71.4

Table 2: Comparison between LLaVA-Video using ForestPrune and existing SOTA MLLMs. Keeping the same amount of input tokens, we use ForestPrune to scale up the input video frames, and achieve obvious performance gains.

Node	FPrune	NExT - QA		VideoMME		MLVU
Node	FPrune	Acc $\uparrow$	Retain $\uparrow$	Acc $\uparrow$	Retain $\uparrow$	Acc $\uparrow$	Retain $\uparrow$
G-Prune	$\times$	74.7	91.0%	57.7	91.9%	59.9	84.0%
VisionZip	$\times$	76.5	91.7%	57.0	90.8%	64.5	90.5%
$\times$	$\checkmark$	80.1	96.0%	58.9	93.8%	63.9	89.6%
G-Prune	$\checkmark$	81.0	97.1%	59.1	94.1%	66.4	93.1%
VisionZip	$\checkmark$	80.6	96.6%	58.4	93.0%	66.7	93.5%

Table 3: Ablation study of the construction manners and numbers of the potential tree nodes

\mathbf{F}_{nod}

for ForestPrune on LLaVA-Video.

Figure 4: Efficiency comparison between ForestPrune and three existing methods. It is using LLaVA-Video with 90% compression ratio. GPU memory records the peak usage.

$\tau_{s}$	$\tau_{p}$	NExT - QA		VideoMME		MLVU
$\tau_{s}$	$\tau_{p}$	Acc $\uparrow$	Retain $\uparrow$	Acc $\uparrow$	Retain $\uparrow$	Acc $\uparrow$	Retain $\uparrow$
0.95	0.80	80.7	97.3%	58.8	93.6%	66.4	93.1%
0.90	0.80	80.9	97.6%	58.8	93.6%	66.4	93.1%
0.80	0.80	79.8	96.3%	58.8	93.6%	66.4	93.1%
0.80	0.50	79.4	95.8%	58.5	93.2%	65.8	92.3%
0.80	0.60	79.5	95.9%	58.5	93.2%	65.3	91.6%

Table 4: Ablation of the semantic and spatial hyper-parameters of ForestPrune on LLaVA-Video-7B, i.e. ,

\tau_{s}

and

\tau_{p}

Figure 5: Visualized results of ForestPrune. Subfigure-(a) shows the spatial-temporal tree built by ForestPrune and the compression results, which show ForestPrune’s global spatial-temporal modeling capabilities. Subfigure-(b) shows the compression results of ForestPrune, G-Prune, and VisionZip, showcasing ForestPrune’s ability to reduce temporal redundancy compared to image compression methods. The ORANGE and BLUE trees are the spatial-temporal trees. Frames with scene changes occur are shown with GREEN boxes.

Explore similar work

Apr 2, 2026

HieraVid: Hierarchical Token Pruning for Fast Video Large Language Models

Video Large Language Models (VideoLLMs) have demonstrated impressive capabilities in video understanding, yet the massive number of input video tokens incurs a significant computational burden for deployment. Existing methods mainly prune video tokens at input level while neglecting the inherent information structure embedded in videos and large language models (LLMs). To address this, we propose HieraVid, a hierarchical pruning framework that progressively and dynamically reduces visual redundancy. Based on two observations that videos possess the segment-frame structure and LLMs internally propagate multi-modal information unidirectionally, we decompose pruning into three levels: 1) segment-level, where video tokens are first temporally segmented and spatially merged; 2) frame-level, where similar frames within the same segment are jointly pruned to preserve diversity; 3) layer-level, redundancy gradually shrinks as LLM layer increases w/o compromising performance. We conduct extensive experiments on four widely used video understanding benchmarks to comprehensively evaluate the effectiveness of HieraVid. Remarkably, with only 30% of tokens retained, HieraVid achieves new state-of-the-art performance, while maintaining over 98% and 99% of the performance of LLaVA-Video-7B and LLaVA-OneVision-7B, respectively.

Yansong Guo, Chaoyang Zhu, Jiayi Ji +2

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China Xiamen University Xiamen, Fujian, China

Token Pruning Video-Language Models Visual Token Pruning Video Token Compression

Ask PDF

Oct 12, 2026

FoveaTree: Information-Density Trees for Query-Guided Visual Token Pruning in Video Large Language Models

Video large language models (Video LLMs) encode each frame into hundreds of visual tokens, making video inference expensive. Existing methods reduce this cost through similarity-based merging or attention-based selection but face two challenges: Hierarchy-blind Pruning, which overlooks regional compressibility and can discard critical evidence, and Attention Drift, where positional effects and attention sinks shift text-to-vision attention away from task relevance. We propose FoveaTree, a two-stage, training-free visual-token pruning framework. First, Merge-Gain Density Estimation measures the feature distortion caused by regional merging and constructs a quadtree-derived density map to guide token dropping, merging, and retention. Second, Attention Tree Pruning uses the surviving units and inherited tree topology to refine text-to-vision attention in intermediate LLM layers, preserving relevant regional evidence while removing low-relevance content. We evaluate FoveaTree on InternVL3.5-8B and Qwen3-VL-8B across five video question-answering benchmarks and one captioning benchmark at 50%, 30%, and 10% visual-token retention. FoveaTree consistently achieves overall state-of-the-art performance across the evaluated settings, offering a strong accuracy-efficiency trade-off.

Zihan Meng, Weidong Chen, Dexiang Hong +2

University of Science and Technology of China · Harbin Institute of Technology

Efficient VLM Inference Visual Token Pruning Large Vision-Language Models Video QA

Ask PDF

Oct 9, 2026

Video Codec-Guided Training-Free Token Compression for Video Large Language Models

Video large language models (VLLMs) rely on long visual token sequences to understand complex temporal content, which makes visual token compression important for efficient inference. Existing training-free compression methods mainly reduce the number of tokens sent to the language model, but the compression process itself can still require many visual feature comparisons, especially for intra-frame token protection and inter-frame temporal token merging. This paper proposes a video codec-guided training-free token compression method for VLLMs that uses coding information already available in compressed video as a low-cost search prior. The method aligns reference-frame dependencies, motion vectors, and residuals with the visual token grid. Residual information forms compact candidate pools for token protection, while motion information is propagated along reference-frame chains to construct localized candidates for temporal token merging between sparsely sampled frames. Local visual feature similarity comparison is then used within these candidates to make the final compression decisions, without modifying the base VLLM. Experiments on three representative VLLMs and four video-understanding benchmarks show that the proposed method achieves performance comparable to the state-of-the-art training-free baseline FlashVID, and in some settings surpasses it, while substantially reducing the feature comparisons required by token protection and temporal matching. These results demonstrate that codec information can effectively improve the accuracy-complexity tradeoff of video token compression.

Xihua Sheng, Ruijie Liu, Chang Wen Chen

Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China

Video Token Compression Video-Language Models VLM Inference

Ask PDF