Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understanding

Wang Chen; Yuhui Zeng; Yongdong Luo; Tianyu Xie; Luojun Lin; Jiayi Ji; Yan Zhang; Xiawu Zheng

cs.CVFeb 28, 2026

Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understanding

Authors: Wang Chen, Yuhui Zeng, Yongdong Luo, Tianyu Xie, Luojun Lin, Jiayi Ji, Yan Zhang, Xiawu Zheng

Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · College of Computer and Data Science, Fuzhou University

Abstract

Frame selection is crucial due to high frame redundancy and limited context windows when applying Large Vision-Language Models (LVLMs) to long videos. Current methods typically select frames with high relevance to a given query, resulting in a disjointed set of frames that disregard the narrative structure of video. In this paper, we introduce Wavelet-based Frame Selection by Detecting Semantic Boundary (WFS-SB), a training-free framework that presents a new perspective: effective video understanding hinges not only on high relevance but, more importantly, on capturing semantic shifts - pivotal moments of narrative change that are essential to comprehending the holistic storyline of video. However, direct detection of abrupt changes in the query-frame similarity signal is often unreliable due to high-frequency noise arising from model uncertainty and transient visual variations. To address this, we leverage the wavelet transform, which provides an ideal solution through its multi-resolution analysis in both time and frequency domains. By applying this transform, we decompose the noisy signal into multiple scales and extract a clean semantic change signal from the coarsest scale. We identify the local extrema of this signal as semantic boundaries, which segment the video into coherent clips. Building on this, WFS-SB comprises a two-stage strategy: first, adaptively allocating a frame budget to each clip based on a composite importance score; and second, within each clip, employing the Maximal Marginal Relevance approach to select a diverse yet relevant set of frames. Extensive experiments show that WFS-SB significantly boosts LVLM performance, e.g., improving accuracy by 5.5% on VideoMME, 9.5% on MLVU, and 6.2% on LongVideoBench, consistently outperforming state-of-the-art methods. Our code is available at https://github.com/MAC-AutoML/WFS-SB.

Figures & tables

Figure 1: A comparison of frame selection strategies, illustrated with the query: “What is the process of applying makeup around the eyes in the video?”. (Top) Current approach: Selects scattered, high-relevance frames (e.g., any frame with eyes), which fails to preserve the procedural order. (Bottom) Our approach: Employs a wavelet-based method to first segment the video into semantically coherent segments (e.g., applying eyeliner, shaping eyebrows) and then samples frames from each segment. This preserves the semantic integrity essential for process comprehension.

Figure 2: An overview of our proposed WFS-SB framework. The process unfolds in three main stages: (1) Wavelet-based Semantic Boundary Identification: The raw query-frame relevance signal is decomposed using a multi-level Discrete Wavelet Transform. We isolate and reconstruct the coarsest detail coefficients to generate a robust semantic change signal, whose peaks define the boundaries of coherent semantic segments. (2) Adaptive Budget Allocation: A composite importance score is computed for each segment, which guides a softmax-weighted distribution of the total frame budget $K$ = 8. (3) Diversity-Aware Intra-segment Selection: Finally, a localized Maximal Marginal Relevance (MMR) selection is performed within each segment to ensure both relevance and diversity in the final keyframe set.

Model	Method	Size	Frame	VideoMME [ 8 ]			MLVU [ 47 ]			LongVideoBench [ 37 ]
Model	Method	Size	Frame	Base	+Method	$\Delta$	Base	+Method	$\Delta$	Base	+Method	$\Delta$
LLaVA-OV [ 10 ]	Frame-Voyager ∗‡ [ 40 ]	7B	8	53.3	57.5	+4.2	58.5	65.6	+7.1	-	-	-
	KFC ∗ [ 6 ]	7B	8	53.3	55.4	+2.1	58.5	66.2	+7.7	54.5	55.6	+1.1
	BOLT ∗ [ 18 ]	7B	8	53.8	56.1	+2.3	58.9	63.4	+4.5	54.2	55.6	+1.4
	AKS † [ 32 ]	7B	8	54.1	58.2	+4.1	58.6	62.9	+4.3	54.2	58.4	+4.2
	FrameOracle ∗‡ [ 11 ]	7B	8	53.8	57.5	+3.7	58.4	62.9	+4.5	54.3	56.0	+1.7

Table 1: Comparison of VLMs and various frame selection methods on VideoMME, MLVU, and LVB. We report accuracy scores (%). Bold indicates best performance and underline indicates second-best performance in each model-dataset group. † Reproduced results, ∗ Reported results, ‡ Training-based methods, § LVLM-dependent iterative method.

Figure 3: Performance comparison across different frame budgets K on VideoMME. WFS-SB consistently outperforms uniform sampling across all LVLMs and budget settings.

Model Scale	Baseline	+WFS-SB	$\Delta$
Qwen2.5-VL-3B	54.1/57.8	58.4/60.0	+4.3 / +2.2
Qwen2.5-VL-7B	57.7/61.2	61.9/64.4	+4.2 / +3.2
Qwen2.5-VL-32B	60.2/62.9	63.0/66.1	+2.8 / +3.2
Qwen2.5-VL-72B	63.3/66.2	66.1/68.9	+2.8 / +2.7

Table 2: Ablation of different LVLM scales using Qwen2.5-VL on VideoMME. Accuracy (%) is reported. Results show performance with

K

=16 frames (left) and

K

=32 frames (right).

\Delta

column shows improvement gains for 16 and 32 frames respectively.

VLM Scorer	VideoMME	MLVU	LVB
Uniform	57.7	56.2	57.0
BLIP-ITM [ 12 ]	62.8	67.7	63.2
CLIP-VIT-B [ 25 ]	61.7	66.5	60.6
SigLIP-so400m [ 41 ]	61.9	66.4	61.9
BLIP-2-ITM (Ours) [ 13 ]	61.9	67.9	62.5

Table 3: Ablation of different VLMs for query-frame similarity scoring with Qwen2.5-VL-7B (

K

=16). Accuracy (%) is reported. Although BLIP-ITM achieves highest performance on VideoMME and LVB, we select BLIP-2-ITM as default because it provides more consistent average benefits across diverse settings.

Configuration	VideoMME	MLVU
Uniform Sampling	57.7	56.2
WFS-SB (Our full method)	61.9	67.9
w/o DWT (use raw local minima)	60.8	64.6
w/o DWT (use raw gradient)	61.2	66.8
w/o Adaptive B.A. (Average B.A.)	61.6	67.4
w/o MMR (topK selection)	60.9	66.7

Table 4: Ablation of different components with Qwen2.5-VL-7B (

K

=16). We remove each component to measure its contribution. Accuracy (%) is reported. B.A. means Budget Allocation.

Wavelet	Db4	Db4	Db8	Haar	Sym4	Bior3.3
$l$	3	4	3	3	3	3
VideoMME	61.9	60.4	61.9	61.3	61.9	61.9
MLVU	67.9	68.3	66.4	68.1	67.6	68.2

Table 5: Ablation of wavelet families and decomposition level

l

with Qwen2.5-VL-7B (

K

=16). Accuracy (%) is reported.

$\boldsymbol{w_{d}},\boldsymbol{w_{a}},\boldsymbol{w_{m}},\boldsymbol{w_{v}}$	$\boldsymbol{\lambda}$	VideoMME	MLVU
0.4, 0.2, 0.3, 0.1	0.5	61.9	67.9
0.0, 0.3, 0.3, 0.2	0.5	61.6	64.8
0.5, 0.0, 0.3, 0.2	0.5	61.8	66.8
0.5, 0.3, 0.0, 0.2	0.5	62.0	66.8
0.5, 0.2, 0.3, 0.0	0.5	62.1	67.4
0.4, 0.2, 0.3, 0.1	0.3	61.2	67.5

Table 6: Ablation of hyperparameters on VideoMME and MLVU with Qwen2.5-VL-7B (

K

=16). Accuracy (%) is reported.

Component	Time (s)	Complexity
ITM Signal Extraction	19.4	$O(N)$
DWT and Boundary Detection	$\sim$ 0	$O(N\log N)$
Budget Allocation	$\sim$ 0	$O(M)$
MMR Selection	0.7	$O(NK^{2})$
LVLM Inference	4.4	-

Table 7: Computational efficiency breakdown on VideoMME with Qwen2.5-VL-7B (avg. N=1040 frames,

K

=32). Times measured on a single NVIDIA A800 GPU and two Intel Xeon Silver 4314 CPUs (16 cores), averaged over 900 videos.

Figure 4: Visualization of our WFS-SB framework on a daily itinerary video. Our wavelet-based method first partitions the video into coherent segments by detecting robust semantic boundaries. Subsequently, after filtering segments based on importance scores, Maximal Marginal Relevance sampling is applied within each to select keyframes that preserve the essential narrative, correctly capturing the sequence of class, tutoring, and self-study.

$\alpha$	$\boldsymbol{\beta}$	VideoMME	MLVU
0.5	0.05	61.9	67.9
0.0	0.05	62.6	67.2
1.0	0.05	61.5	68.4
0.5	0.00	61.8	67.6
0.5	0.1	62.0	67.9

Table 8: Robustness of peak detection to hyperparameter variations. Evaluated on VideoMME and MLVU with Qwen2.5-VL-7B (

K

=16). Performance varies by only 1.1% or 1.2% across different settings, demonstrating strong hyperparameter insensitivity.

Model

Method

Size

Frame

VideoMME [ 8 ]

MLVU [ 47 ]

LongVideoBench [ 37 ]

Base

+Method

\Delta

Base

+Method

\Delta

Base

+Method

\Delta

Qwen2-VL-7B [ 35 ]

KFC [ 6 ]

56.7

+1.7

59.6

65.9

+6.3

53.4

54.6

+1.2

Q-Frame [ 45 ]

8 (4+8+32) ∗

53.7

58.3

+4.6

56.9

65.4

+8.5

53.5

58.4

+4.9

WFS-SB

52.0

56.4

+4.4

55.0

63.4

+8.4

51.7

56.5

+3.8

Table 9: Extended comparison with additional baseline methods on VideoMME, MLVU, and LongVideoBench using Qwen2-VL-7B (

K

=8). WFS-SB achieves competitive performance with consistent improvements across all benchmarks (+4.4%, +8.4%, +3.8%). ∗ Same token budget but more frames.

Figure 5: Performance across different frame budgets on MLVU. WFS-SB consistently outperforms uniform sampling across all LVLMs and budget settings.

Figure 6: Performance across different frame budgets on LongVideoBench. WFS-SB demonstrates robust improvements across multiple architectures and budget constraints.

Sampling Rate	8 F.	16 F.	32 F.	ITM Time (s)
Uniform Sampling	53.2	57.7	61.2	-
1 fps	59.3	61.9	64.4	19.4
1-0.75-0.5 fps	58.9	62.0	64.5	10.5
1-0.5-0.25 fps	59.4	61.9	64.6	5.8

Table 10: Impact of adaptive FPS sampling on performance and ITM extraction time. Evaluated on VideoMME with Qwen2.5-VL-7B. The notation ”1-0.75-0.5” indicates using 1 fps for short videos, 0.75 fps for medium videos, and 0.5 fps for long videos. Adaptive strategies reduce ITM time by 46% (1-0.75-0.5) and 70% (1-0.5-0.25) respectively while maintaining or improving performance. F. represents the number of input frames.

VLM Scorer	VideoMME	MLVU	LVB
Uniform	60.6	60.9	57.2
BLIP-ITM [ 12 ]	63.2	69.5	61.8
CLIP-VIT-B [ 25 ]	63.6	67.3	62.2
SigLIP-so400m [ 41 ]	62.4	68.9	61.5
BLIP-2-ITM (Ours) [ 13 ]	64.3	71.0	62.4

Table 11: Ablation on different VLMs for query-frame similarity scoring with LLaVA-Video-7B (

K

=16). We select BLIP-2-ITM as default for its superior performance across all benchmarks.

$\boldsymbol{\eta}$	VideoMME	MLVU
1.2	61.9	67.9
-	61.5	67.2
1.0	61.7	67.5
1.5	61.9	68.1

Table 12: Robustness of segment filtering to hyperparameter

\eta

. Evaluated on VideoMME and MLVU with Qwen2.5-VL-7B. Performance varies by only 0.2-0.6% across different

\eta

values. Filtering is effective: compare ”-” (no filtering, all segments retained) vs. others.

Figure 7: Qualitative comparison on Ellora Caves video. Query: ”Which of the following things can not be found in Ellora Caves? A. A shiva statue. B. An elephant statue. C. A monument about Ramayana. D. Piles of stone.” Top: Uniform sampling (Answer: C) - Too random, lacks query-specific awareness, missing critical frames. Middle: Top-K sampling (Answer: A) - Focuses on highly salient objects (e.g., elephant statue) but misses other equally important yet less visually prominent frames. Bottom: Our WFS-SB (Answer: D, correct) - Through wavelet-based boundary detection, effectively extracts semantic boundaries to segment the video. Adaptive budget allocation and intra-segment diversity selection successfully identify keyframes that enable correct question answering.

Explore similar work

Event-Anchored Frame Selection for Effective Long-Video Understanding
Mar 1, 2026Wang Chen, Yongdong Luo, Yuhui Zeng +5Video Frame Selection Query-Aware Video Frame Selection
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · College of Computer and Data Science, Fuzhou University
Adaptive Greedy Frame Selection for Long Video Understanding
Mar 20, 2026Yuning Huang, Xiaoyu Ji, Joseph Huang +2Query-Aware Video Frame Selection Long-Video QA
Purdue University, West Lafayette, IN 47906
RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding
Aug 30, 2026Shanqing Xu, Meng Luo, Mengchen Qian +7Large Vision-Language Models Long-Video Understanding
Huazhong University of Science and Technology · National University of Singapore · Fudan University

Mar 1, 2026J/K move · Enter open · S save

Event-Anchored Frame Selection for Effective Long-Video Understanding

Wang Chen, Yongdong Luo, Yuhui Zeng, Luojun Lin+4

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · College of Computer and Data Science, Fuzhou University

Video Frame Selection Query-Aware Video Frame Selection Temporal Video Understanding Long-Context VLMs

Open paper AskPDF arXiv

Massive frame redundancy and limited context window make efficient frame selection crucial for long-video understanding with large vision-language models (LVLMs). Prevailing approaches, however, adopt a flat sampling paradigm which treats the video as an unstructured collection of frames. In this paper, we introduce Event-Anchored Frame Selection (EFS), a hierarchical, event-aware pipeline. Leveraging self-supervised DINO embeddings, EFS first partitions the video stream into visually homogeneous temporal segments, which serve as proxies for semantic events. Within each event, it then selects the most query-relevant frame as an anchor. These anchors act as structural priors that guide a global refinement stage using an adaptive Maximal Marginal Relevance (MMR) scheme. This pipeline ensures the final keyframe set jointly optimizes for event coverage, query relevance, and visual diversity. As a training-free, plug-and-play module, EFS can be seamlessly integrated into off-the-shelf LVLMs, yielding substantial gains on challenging video understanding benchmarks. Specifically, when applied to LLaVA-Video-7B, EFS improves accuracy by 4.7%, 4.9%, and 8.8% on VideoMME, LongVideoBench, and MLVU, respectively.

Abstract

Explore similar work

Event-Anchored Frame Selection for Effective Long-Video Understanding

Adaptive Greedy Frame Selection for Long Video Understanding

RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding