Event-Anchored Frame Selection for Effective Long-Video Understanding

Wang Chen; Yongdong Luo; Yuhui Zeng; Luojun Lin; Tianyu Xie; Fei Chao; Rongrong Ji; Xiawu Zheng

cs.CVMar 1, 2026

Event-Anchored Frame Selection for Effective Long-Video Understanding

Authors: Wang Chen, Yongdong Luo, Yuhui Zeng, Luojun Lin, Tianyu Xie, Fei Chao, Rongrong Ji, Xiawu Zheng

Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · College of Computer and Data Science, Fuzhou University

Abstract

Massive frame redundancy and limited context window make efficient frame selection crucial for long-video understanding with large vision-language models (LVLMs). Prevailing approaches, however, adopt a flat sampling paradigm which treats the video as an unstructured collection of frames. In this paper, we introduce Event-Anchored Frame Selection (EFS), a hierarchical, event-aware pipeline. Leveraging self-supervised DINO embeddings, EFS first partitions the video stream into visually homogeneous temporal segments, which serve as proxies for semantic events. Within each event, it then selects the most query-relevant frame as an anchor. These anchors act as structural priors that guide a global refinement stage using an adaptive Maximal Marginal Relevance (MMR) scheme. This pipeline ensures the final keyframe set jointly optimizes for event coverage, query relevance, and visual diversity. As a training-free, plug-and-play module, EFS can be seamlessly integrated into off-the-shelf LVLMs, yielding substantial gains on challenging video understanding benchmarks. Specifically, when applied to LLaVA-Video-7B, EFS improves accuracy by 4.7%, 4.9%, and 8.8% on VideoMME, LongVideoBench, and MLVU, respectively.

Figures & tables

Figure 1: A conceptual illustration of our Event-anchored Frame Selection (EFS) in contrast to the flat sampling baseline. Flat sampling often yields redundant frames and misses key events, causing LVLMs to fail, while EFS partitions videos into events, selects query-relevant anchors, and performs anchor-guided global refinement to enable correct chronological reasoning.

Figure 2: An illustration of our Event-Anchored Frame Selection (EFS) pipeline. Given a user query, such as “How many times does the instructor appear in different scenarios?”, EFS operates in distinct stages: (1) It first acquires visual signals (from DinoV2) and semantic relevance scores (from BLIP2-ITM). (2) The visual signals are used to partition the video into coherent events at local minima of temporal similarity. (3) Within each event, it localizes the most query-relevant frame as an anchor. (4) This initial set of anchors is then globally refined via Adaptive MMR to form the final keyframe set. (5) Finally, this curated set enables the Large Vision-Language Models to accurately answer the query, correctly identifying the four appearances.

Proprietary LVLMs
Model	Params	Frames	VideoMME (w.o. sub.)				LongVideo	MLVU
Model	Params	Frames	Short	Medium	Long	Overall	Bench (val)	(m-avg)
GPT-4o mini [ 25 ]	-	250	72.5	63.1	58.6	64.8	-	-
GPT-4o [ 26 ]	-	384/256/0.5fps	80.0	70.3	65.3	71.9	66.7	64.6
Open-Source LVLMs
Video-LLaVA [ 14 ]	7B	8	45.3	38.0	36.2	39.9	39.1	47.3

Table 1: Video-based question answering accuracy (%) of previous LVLMs on Video-MME [ 9 ] , LongVideoBench [ 37 ] and MLVU [ 45 ] . Our EFS is applied to three off-the-shelf LVLMs. Frames and LLM indicate the number of video frames input to the LVLMs and the number of parameters in the LLM part, respectively.

Figure 3: A quantitative comparison on VideoMME using LLaVA-Video-7B, where EFS consistently delivers the largest accuracy gains across 8/16/32/64-frame budgets over different sampling strategies. The text to the left of each bar represents the baseline performance for that method, while the text on the right indicates the corresponding gain.

Figure 4: Video-based question answering accuracy (%) with different numbers of input frames.

Methods		VideoMME	LVB
Uniform Sampling		60.6	57.2
Random Partition		62.9	59.8
Clustered Partition		63.3	60.7
Random Initialization		62.1	60.3
Centralized Initialization		61.9	60.1
MMR w.o. Adaptive	$\tau$ =0.4	63.7	59.9

Table 2: Video-based question answering accuracy (%) of different Strategies in EFS. We compare the performance of our final method against variants with different strategies for event partitioning, anchor initialization, and the refinement process on the VideoMME and LVB benchmarks.

Method	VideoMME	LVB
Uniform	60.0	57.2
Adaptive Detectors [ 4 ]	63.6	60.3
Content Detectors [ 4 ]	63.6	60.4
Hash Detectors [ 4 ]	62.8	60.4
Histogram Detectors [ 4 ]	62.7	60.3
Threshold Detectors [ 4 ]	63.2	59.8

Table 3: Video-based question answering accuracy (%) using different shot boundary detectors on LLaVA-Video-7B.

Methods	Frames	VideoMME	LVB
Uniform	8/16	56.2/60.6	54.9/57.2
CLIP + CLIP	8/16	60.2/63.1	57.6/60.2
DINOv2 + CLIP	8/16	60.5/63.1	58.4/60.3
CLIP + BLIP2-ITM	8/16	60.4/63.3	58.8/60.4
DINOv2 + BLIP2-ITM	8/16	60.9/64.0	59.8/61.3

Table 4: Ablation of signal acquisition. Left of “+” denotes the visual feature extractor for event detection, while the right side is the model for query-frame relevance scoring.

Panel of Table 5 — Table 5: Ablation study on the hyper-parameters $M$ (maximum number of events) and $\alpha$ (threshold relaxation factor). All experiments are conducted on the VideoMME using LLaVA-Video-7B.

Figure 5: Qualitative comparison of our EFS versus the uniform sampling. The uniform baseline fails by missing key events, while EFS explicitly identifies the video’s underlying event structure, enabling accurate reasoning and correct answers. Please zoom in for details.

Preprocessing Time (seconds):
Component	Short	Medium	Long	Overall
Video Reading	1.45	9.40	48.94	19.93
Vision Signal	1.16	7.45	33.75	14.12
Semantic Signal	1.34	8.48	40.75	16.85
EFS Selection	0.03	0.16	0.95	0.38
Total Preprocess (s)	4.0	25.5	124.4	51.3

Table 6: Computational cost and accuracy comparison on VideoMME with LLaVA-Video-7B. We report average per-video time (seconds) for each component. All experiments conducted on a single NVIDIA A800 GPU with two Intel Xeon Silver 4314 CPUs (16 cores), at 1 fps with 16 input frames.

Figure 6: Detailed performance comparison of our method (EFS) against the baseline across all 12 subtasks of the Video-MME benchmark [ 9 ] . The results demonstrate that our EFS approach consistently surpasses the baseline in every category, underscoring the broad effectiveness of our method in enhancing diverse video understanding capabilities.

Method	VideoMME	LVB (val)	MLVU
uniform	60.6	57.2	60.9
EFS( $l$ =3)	64.0	61.3	69.0
EFS( $l$ =5)	63.3	61.7	68.0
EFS( $l$ =7)	63.4	60.9	68.6

Table 7: Ablation on the window size

l

. We report the video-based question answering accuracy (%) on the VideoMME, LVB, and MLVU benchmarks. The best results are highlighted in bold .

Panel of Table 8 — Table 8: Ablation of $M$ and $\alpha$ . Results are shown as accuracy (%) on LongVideoBench/MLVU.

Methods	VideoMME	LVB	MLVU
LLaVA-Video (Uniform)	60.6	57.2	60.9
LLaVA-Video + EFS	64.0	61.3	69.0
LLaVA-Video + part EFS	62.5 (-1.5)	60.7 (-0.6)	68.6 (-0.4)

Table 9: Performance comparison of the CoT-based adaptive sampling strategy. The performance drop in “part EFS” highlights the universal effectiveness of EFS, even on general queries where the system reverted to uniform sampling.

Figure 7: T-SNE visualization of DINOv2 features for sampled video frames. Each point represents one frame, and color indicates the event predicted by our EFS pipeline.

Explore similar work

EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understanding
Oct 6, 2026Yuhao Qin, Junbo Wang, Yuke Li +1Multimodal RAG Video-Language Models
Northwestern Polytechnical University Xi’an, China
Gaussian Mixture Modeling for Event-Aware Visual Allocation in Long Video Understanding
Jul 14, 2026Yifan Lu, Ziqi Zhang, Chunfeng Yuan +3Large Vision-Language Models Video Frame Selection
Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information, CASIA · State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences +2
RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding
Aug 30, 2026Shanqing Xu, Meng Luo, Mengchen Qian +7Large Vision-Language Models Long-Video Understanding
Huazhong University of Science and Technology · National University of Singapore · Fudan University

Oct 6, 2026J/K move · Enter open · S save

EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understanding

Yuhao Qin, Junbo Wang, Yuke Li, Yining Zhu

Northwestern Polytechnical University Xi’an, China

Multimodal RAG Video-Language Models Long-Video Understanding Temporal Video Understanding

Open paper AskPDF arXiv

Current large video-language models (LVLMs) still face challenges when dealing with long videos, mainly because frames are often processed independently, making it difficult to capture temporal dependencies across events. Although retrieval-augmented approaches have been introduced to provide additional context, most of them operate at the frame or snippet level, which limits their ability to model how events evolve over time and relate to each other. In this paper, we propose Event Chain Retrieval-Augmented Generation (EC-RAG), a training-free framework that organizes video content into an explicit event chain before question answering. Instead of retrieving isolated frames or text segments, EC-RAG first partitions the video into semantically coherent segments, represents each segment using multi-modal signals, and then links them into a structured chain that preserves temporal order and captures inter-event relationships. Given a query, the system identifies relevant events within this chain and gathers supporting evidence from the associated modalities. Our approach offers several practical advantages: (i) event-level abstraction that better reflects how video content is naturally structured, enabling more reliable localization compared to frame-level retrieval; (ii) structured multi-modal fusion that aggregates speech, text, and visual cues at the event level, allowing complementary information to be more effectively utilized during reasoning; and (iii) plug-and-play compatibility with existing LVLM backbones, requiring no additional training or reliance on proprietary models. Experiments on Video-MME, MLVU, and LongVideoBench show that this event-centric design consistently outperforms frame-level retrieval baselines, highlighting the importance of modeling temporal structure for long-video understanding.

Abstract

Explore similar work

EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understanding

Gaussian Mixture Modeling for Event-Aware Visual Allocation in Long Video Understanding

RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding