cs.CVFeb 28, 2026

Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understanding

Authors: Wang Chen, Yuhui Zeng, Yongdong Luo, Tianyu Xie, Luojun Lin, Jiayi Ji, Yan Zhang, Xiawu Zheng

Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · College of Computer and Data Science, Fuzhou University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Frame selection is crucial due to high frame redundancy and limited context windows when applying Large Vision-Language Models (LVLMs) to long videos. Current methods typically select frames with high relevance to a given query, resulting in a disjointed set of frames that disregard the narrative structure of video. In this paper, we introduce Wavelet-based Frame Selection by Detecting Semantic Boundary (WFS-SB), a training-free framework that presents a new perspective: effective video understanding hinges not only on high relevance but, more importantly, on capturing semantic shifts - pivotal moments of narrative change that are essential to comprehending the holistic storyline of video. However, direct detection of abrupt changes in the query-frame similarity signal is often unreliable due to high-frequency noise arising from model uncertainty and transient visual variations. To address this, we leverage the wavelet transform, which provides an ideal solution through its multi-resolution analysis in both time and frequency domains. By applying this transform, we decompose the noisy signal into multiple scales and extract a clean semantic change signal from the coarsest scale. We identify the local extrema of this signal as semantic boundaries, which segment the video into coherent clips. Building on this, WFS-SB comprises a two-stage strategy: first, adaptively allocating a frame budget to each clip based on a composite importance score; and second, within each clip, employing the Maximal Marginal Relevance approach to select a diverse yet relevant set of frames. Extensive experiments show that WFS-SB significantly boosts LVLM performance, e.g., improving accuracy by 5.5% on VideoMME, 9.5% on MLVU, and 6.2% on LongVideoBench, consistently outperforming state-of-the-art methods. Our code is available at https://github.com/MAC-AutoML/WFS-SB.

Figures & tables

Figure 1: A comparison of frame selection strategies, illustrated with the query: “What is the process of applying makeup around the eyes in the video?”. (Top) Current approach: Selects scattered, high-relevance frames (e.g., any frame with eyes), which fails to preserve the procedural order. (Bottom) Our approach: Employs a wavelet-based method to first segment the video into semantically coherent segments (e.g., applying eyeliner, shaping eyebrows) and then samples frames from each segment. This preserves the semantic integrity essential for process comprehension.
Figure 2: An overview of our proposed WFS-SB framework. The process unfolds in three main stages: (1) Wavelet-based Semantic Boundary Identification: The raw query-frame relevance signal is decomposed using a multi-level Discrete Wavelet Transform. We isolate and reconstruct the coarsest detail coefficients to generate a robust semantic change signal, whose peaks define the boundaries of coherent semantic segments. (2) Adaptive Budget Allocation: A composite importance score is computed for each segment, which guides a softmax-weighted distribution of the total frame budget KK = 8. (3) Diversity-Aware Intra-segment Selection: Finally, a localized Maximal Marginal Relevance (MMR) selection is performed within each segment to ensure both relevance and diversity in the final keyframe set.
ModelMethodSizeFrameVideoMME [ 8 ]MLVU [ 47 ]LongVideoBench [ 37 ]
Base+MethodΔ\DeltaBase+MethodΔ\DeltaBase+MethodΔ\Delta
LLaVA-OV [ 10 ]Frame-Voyager ∗‡ [ 40 ]7B853.357.5+4.258.565.6+7.1---
KFC ∗ [ 6 ]7B853.355.4+2.158.566.2+7.754.555.6+1.1
BOLT ∗ [ 18 ]7B853.856.1+2.358.963.4+4.554.255.6+1.4
AKS † [ 32 ]7B854.158.2+4.158.662.9+4.354.258.4+4.2
FrameOracle ∗‡ [ 11 ]7B853.857.5+3.758.462.9+4.554.356.0+1.7
Table 1: Comparison of VLMs and various frame selection methods on VideoMME, MLVU, and LVB. We report accuracy scores (%). Bold indicates best performance and underline indicates second-best performance in each model-dataset group. † Reproduced results, ∗ Reported results, ‡ Training-based methods, § LVLM-dependent iterative method.
Figure 3: Performance comparison across different frame budgets K on VideoMME. WFS-SB consistently outperforms uniform sampling across all LVLMs and budget settings.
Model ScaleBaseline+WFS-SBΔ\Delta
Qwen2.5-VL-3B54.1/57.858.4/60.0+4.3 / +2.2
Qwen2.5-VL-7B57.7/61.261.9/64.4+4.2 / +3.2
Qwen2.5-VL-32B60.2/62.963.0/66.1+2.8 / +3.2
Qwen2.5-VL-72B63.3/66.266.1/68.9+2.8 / +2.7
Table 2: Ablation of different LVLM scales using Qwen2.5-VL on VideoMME. Accuracy (%) is reported. Results show performance with KK =16 frames (left) and KK =32 frames (right). Δ\Delta column shows improvement gains for 16 and 32 frames respectively.
VLM ScorerVideoMMEMLVULVB
Uniform57.756.257.0
BLIP-ITM [ 12 ]62.867.763.2
CLIP-VIT-B [ 25 ]61.766.560.6
SigLIP-so400m [ 41 ]61.966.461.9
BLIP-2-ITM (Ours) [ 13 ]61.967.962.5
Table 3: Ablation of different VLMs for query-frame similarity scoring with Qwen2.5-VL-7B ( KK =16). Accuracy (%) is reported. Although BLIP-ITM achieves highest performance on VideoMME and LVB, we select BLIP-2-ITM as default because it provides more consistent average benefits across diverse settings.
ConfigurationVideoMMEMLVU
Uniform Sampling57.756.2
WFS-SB (Our full method)61.967.9
w/o DWT (use raw local minima)60.864.6
w/o DWT (use raw gradient)61.266.8
w/o Adaptive B.A. (Average B.A.)61.667.4
w/o MMR (topK selection)60.966.7
Table 4: Ablation of different components with Qwen2.5-VL-7B ( KK =16). We remove each component to measure its contribution. Accuracy (%) is reported. B.A. means Budget Allocation.
WaveletDb4Db4Db8HaarSym4Bior3.3
ll343333
VideoMME61.960.461.961.361.961.9
MLVU67.968.366.468.167.668.2
Table 5: Ablation of wavelet families and decomposition level ll with Qwen2.5-VL-7B ( KK =16). Accuracy (%) is reported.
wd,wa,wm,wv\boldsymbol{w_{d}},\boldsymbol{w_{a}},\boldsymbol{w_{m}},\boldsymbol{w_{v}}λ\boldsymbol{\lambda}VideoMMEMLVU
0.4, 0.2, 0.3, 0.10.561.967.9
0.0, 0.3, 0.3, 0.20.561.664.8
0.5, 0.0, 0.3, 0.20.561.866.8
0.5, 0.3, 0.0, 0.20.562.066.8
0.5, 0.2, 0.3, 0.00.562.167.4
0.4, 0.2, 0.3, 0.10.361.267.5
Table 6: Ablation of hyperparameters on VideoMME and MLVU with Qwen2.5-VL-7B ( KK =16). Accuracy (%) is reported.
ComponentTime (s)Complexity
ITM Signal Extraction19.4O(N)O(N)
DWT and Boundary Detection∼\sim 0O(Nlog⁡N)O(N\log N)
Budget Allocation∼\sim 0O(M)O(M)
MMR Selection0.7O(NK2)O(NK^{2})
LVLM Inference4.4-
Table 7: Computational efficiency breakdown on VideoMME with Qwen2.5-VL-7B (avg. N=1040 frames, KK =32). Times measured on a single NVIDIA A800 GPU and two Intel Xeon Silver 4314 CPUs (16 cores), averaged over 900 videos.
Figure 4: Visualization of our WFS-SB framework on a daily itinerary video. Our wavelet-based method first partitions the video into coherent segments by detecting robust semantic boundaries. Subsequently, after filtering segments based on importance scores, Maximal Marginal Relevance sampling is applied within each to select keyframes that preserve the essential narrative, correctly capturing the sequence of class, tutoring, and self-study.
α\alphaβ\boldsymbol{\beta}VideoMMEMLVU
0.50.0561.967.9
0.00.0562.667.2
1.00.0561.568.4
0.50.0061.867.6
0.50.162.067.9
Table 8: Robustness of peak detection to hyperparameter variations. Evaluated on VideoMME and MLVU with Qwen2.5-VL-7B ( KK =16). Performance varies by only 1.1% or 1.2% across different settings, demonstrating strong hyperparameter insensitivity.
ModelMethodSizeFrameVideoMME [ 8 ]MLVU [ 47 ]LongVideoBench [ 37 ]
Base+MethodΔ\DeltaBase+MethodΔ\DeltaBase+MethodΔ\Delta
Qwen2-VL-7B [ 35 ]KFC [ 6 ]7B85556.7+1.759.665.9+6.353.454.6+1.2
Q-Frame [ 45 ]7B8 (4+8+32) ∗53.758.3+4.656.965.4+8.553.558.4+4.9
WFS-SB7B852.056.4+4.455.063.4+8.451.756.5+3.8
Table 9: Extended comparison with additional baseline methods on VideoMME, MLVU, and LongVideoBench using Qwen2-VL-7B ( KK =8). WFS-SB achieves competitive performance with consistent improvements across all benchmarks (+4.4%, +8.4%, +3.8%). ∗ Same token budget but more frames.
Figure 5: Performance across different frame budgets on MLVU. WFS-SB consistently outperforms uniform sampling across all LVLMs and budget settings.
Figure 6: Performance across different frame budgets on LongVideoBench. WFS-SB demonstrates robust improvements across multiple architectures and budget constraints.
Sampling Rate8 F.16 F.32 F.ITM Time (s)
Uniform Sampling53.257.761.2-
1 fps59.361.964.419.4
1-0.75-0.5 fps58.962.064.510.5
1-0.5-0.25 fps59.461.964.65.8
Table 10: Impact of adaptive FPS sampling on performance and ITM extraction time. Evaluated on VideoMME with Qwen2.5-VL-7B. The notation ”1-0.75-0.5” indicates using 1 fps for short videos, 0.75 fps for medium videos, and 0.5 fps for long videos. Adaptive strategies reduce ITM time by 46% (1-0.75-0.5) and 70% (1-0.5-0.25) respectively while maintaining or improving performance. F. represents the number of input frames.
VLM ScorerVideoMMEMLVULVB
Uniform60.660.957.2
BLIP-ITM [ 12 ]63.269.561.8
CLIP-VIT-B [ 25 ]63.667.362.2
SigLIP-so400m [ 41 ]62.468.961.5
BLIP-2-ITM (Ours) [ 13 ]64.371.062.4
Table 11: Ablation on different VLMs for query-frame similarity scoring with LLaVA-Video-7B ( KK =16). We select BLIP-2-ITM as default for its superior performance across all benchmarks.
η\boldsymbol{\eta}VideoMMEMLVU
1.261.967.9
-61.567.2
1.061.767.5
1.561.968.1
Table 12: Robustness of segment filtering to hyperparameter η\eta . Evaluated on VideoMME and MLVU with Qwen2.5-VL-7B. Performance varies by only 0.2-0.6% across different η\eta values. Filtering is effective: compare ”-” (no filtering, all segments retained) vs. others.
Figure 7: Qualitative comparison on Ellora Caves video. Query: ”Which of the following things can not be found in Ellora Caves? A. A shiva statue. B. An elephant statue. C. A monument about Ramayana. D. Piles of stone.” Top: Uniform sampling (Answer: C) - Too random, lacks query-specific awareness, missing critical frames. Middle: Top-K sampling (Answer: A) - Focuses on highly salient objects (e.g., elephant statue) but misses other equally important yet less visually prominent frames. Bottom: Our WFS-SB (Answer: D, correct) - Through wavelet-based boundary detection, effectively extracts semantic boundaries to segment the video. Adaptive budget allocation and intra-segment diversity selection successfully identify keyframes that enable correct question answering.

Explore similar work

CardsList
Mar 1, 2026

Event-Anchored Frame Selection for Effective Long-Video Understanding

Massive frame redundancy and limited context window make efficient frame selection crucial for long-video understanding with large vision-language models (LVLMs). Prevailing approaches, however, adopt a flat sampling paradigm which treats the video as an unstructured collection of frames. In this paper, we introduce Event-Anchored Frame Selection (EFS), a hierarchical, event-aware pipeline. Leveraging self-supervised DINO embeddings, EFS first partitions the video stream into visually homogeneous temporal segments, which serve as proxies for semantic events. Within each event, it then selects the most query-relevant frame as an anchor. These anchors act as structural priors that guide a global refinement stage using an adaptive Maximal Marginal Relevance (MMR) scheme. This pipeline ensures the final keyframe set jointly optimizes for event coverage, query relevance, and visual diversity. As a training-free, plug-and-play module, EFS can be seamlessly integrated into off-the-shelf LVLMs, yielding substantial gains on challenging video understanding benchmarks. Specifically, when applied to LLaVA-Video-7B, EFS improves accuracy by 4.7%, 4.9%, and 8.8% on VideoMME, LongVideoBench, and MLVU, respectively.

Wang Chen, Yongdong Luo, Yuhui Zeng +5

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · College of Computer and Data Science, Fuzhou University

Video Frame SelectionQuery-Aware Video Frame SelectionTemporal Video UnderstandingLong-Context VLMs
AskPDF
Mar 20, 2026

Adaptive Greedy Frame Selection for Long Video Understanding

Large vision--language models (VLMs) are increasingly applied to long-video question answering, yet inference is often bottlenecked by the number of input frames and resulting visual tokens. Naive sparse sampling can miss decisive moments, while purely relevance-driven selection frequently collapses onto near-duplicate frames and sacrifices coverage of temporally distant evidence. We propose a question-adaptive greedy frame selection method that jointly optimizes query relevance and semantic representativeness under a fixed frame budget. Our approach constructs a 1~FPS candidate pool (capped at 1000) with exact timestamp alignment, embeds candidates in two complementary spaces (SigLIP for question relevance and DINOv2 for semantic similarity), and selects frames by greedily maximizing a weighted sum of a modular relevance term and a facility-location coverage term. This objective is normalized, monotone, and submodular, yielding a standard (1-1/e) greedy approximation guarantee. To account for question-dependent trade-offs between relevance and coverage, we introduce four preset strategies and a lightweight text-only question-type classifier that routes each query to its best-performing preset. Experiments on MLVU show consistent accuracy gains over uniform sampling and a strong recent baseline across frame budgets, with the largest improvements under tight budgets.

Yuning Huang, Xiaoyu Ji, Joseph Huang +2

Purdue University, West Lafayette, IN 47906

Query-Aware Video Frame SelectionLong-Video QALarge Vision-Language Models
AskPDF
Aug 30, 2026

RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding

Long videos contain far more visual content than Large Vision-Language Models (LVLMs) can process under a fixed visual-token budget, making frame selection essential. Existing query-aware selectors usually estimate frame-query relevance and build a compact subset from high-scoring frames. Although their mechanisms differ, the similarity sequence is still often treated primarily as values to rank or sample from, rather than as an ordered signal whose shape reflects how query-relevant evidence emerges, peaks, and fades over time. This can obscure frames that explain, contextualize, or follow an event, because such evidence may lie on the rising or falling sides of a nearby relevance peak and receive lower absolute scores. We propose RIDGE, a frame selection framework that reads the frame-query similarity curve as a temporal signal. By using local changes and curvature, RIDGE partitions the timeline into structural regions and applies region-specific selection to preserve event cores, transitions, buildup, aftermath, and contextual frames under a fixed budget. It is a lightweight post-processing step on precomputed frame-query scores and requires neither training nor iterative LVLM calls. Across four long-video benchmarks and three backbones, RIDGE achieves the best performance in most settings and remains competitive in the others.

Shanqing Xu, Meng Luo, Mengchen Qian +7

Huazhong University of Science and Technology · National University of Singapore · Fudan University

Large Vision-Language ModelsLong-Video UnderstandingVideo Frame SelectionQuery-Aware Video Frame Selection
AskPDF