Shot-Aware Frame Sampling for Video Understanding

Mengyu Zhao; Di Fu; Yongyu Xie; Jiaxing Zhang; Zhigang Yuan; Shirin Jalali; Yong Cao

cs.CVMar 18, 2026

Shot-Aware Frame Sampling for Video Understanding

Authors: Mengyu Zhao, Di Fu, Yongyu Xie, Jiaxing Zhang, Zhigang Yuan, Shirin Jalali, Yong Cao

Organizations: ByteDance · Rutgers University · Georgia Tech

Abstract

Video frame sampling is essential for efficient long-video understanding with Vision-Language Models (VLMs), since dense inputs are costly and often exceed context limits. Yet when only a small number of frames can be retained, existing samplers often fail to balance broad video coverage with brief but critical events, which can lead to unreliable downstream predictions. To address this issue, we present InfoShot, a task-agnostic, shot-aware frame sampler for long-video understanding. InfoShot first partitions a video into semantically consistent shots, and then selects two complementary keyframes from each shot: one to represent the main content and one to capture unusual within-shot changes. This design is guided by an information-theoretic objective that encourages the sampled set to retain high information about both shot structure and sparse within-shot deviations. In this way, it improves the chance of preserving both overall video context and short decision-critical moments without requiring any retraining. To better evaluate such short-lived events, we further introduce SynFlash, a synthetic benchmark with controllable sub-second anomaly patterns and frame-level ground truth, and we also evaluate InfoShot on existing anomaly datasets and general video understanding tasks. Experiments show that InfoShot improves anomaly hit rate and downstream Video-QA accuracy under frame number constraints, while matching or outperforming strong baselines on standard video understanding benchmarks.

Figures & tables

Figure 1: The overall framework of our approach. We propose a task-agnostic and plug-and-play sampler, InfoShot, before the VLM to select keyframes under a fixed budget. InfoShot performs greedy shot segmentation in deep feature space (Sec. 3.4 ) and then selects two frames per shot: a common and a unique high-deviation frames (Sec. 3.5 ). The downstream VLM uses the same prompt; only the sampled frame set differs.

Figure 2: Overview of the SynFlash generation pipeline. ImageNet-1k [ 27 ] images are injected at random times into videos formed by concatenating three Panda-70M [ 7 ] clips, creating four transient anomaly types with frame-level ground truth.

Figure 3: Case study from SynFlash (PiP). Left: sampled frames from different methods. Right: example questions.

Metric	Method	Camou.	High-Sat.	Curtain	PiP	Avg.
R	Uniform	11.20	10.00	19.15	11.20	11.81
	VSUMM	74.50	9.15	93.75	9.35	46.54
	TransNet	95.55	16.55	81.15	10.30	46.28
	InfoShot	98.35	87.05	88.45	64.95	84.70
AR	Uniform	29.60	29.05	34.70	29.75	30.78
	VSUMM	56.75	28.40	74.95	29.45	47.39

Table 1: Performance comparison on SynFlash. Metrics include frame-level Recall (R, %), Anomaly Retrieval Accuracy (AR, %), and Temporal Ordering Accuracy (TO, %). Best results are highlighted in bold , and second-best are underlined .

Figure 4: Feature-space distortion

\mathrm{Dist}(\mathcal{K})

(lower is better) on SynFlash.

Method	UCF-Crime		XD-Violence		Weighted Avg.
Method	ER	CR	ER	CR	ER	CR
TransNet	40.97	37.50	37.16	39.00	37.56	38.69
VSUMM	65.97	62.50	57.35	51.60	58.25	53.82
Uniform	69.44	65.62	59.53	54.20	60.56	56.53
InfoShot	69.44	66.41	63.09	58.40	63.75	60.03

Table 2: Event coverage on HIVAU-70k annotations under 0.1 fps. ER: event recall, CR: complete coverage rate. Weighted Avg. is weighted by the number of events/videos in each dataset. Best results are highlighted in bold , and second-best are underlined .

Method	Short	Med.	Avg.
TransNet	65.8	60.4	63.2
VSUMM	67.8	62.5	65.2
Uniform	71.8	62.2	67.0
InfoShot	71.5	62.4	67.0

Table 3: Video-MME results under the same frame budget. Best results are in bold , and second-best are underlined

Figure 5: Similarity matrices on SynFlash under different feature extractors. The first row uses HSV color-histogram features and the second row uses semantic features. Each column shows one example video from a SynFlash subset (Camouflage, High-Sat, Curtain, and PiP). Red arrows indicate the temporal injected flash event.

Metric	Method	Camou.	High-Sat.	Curtain	PiP	Avg.
R	Histogram	93.65	27.05	85.10	40.95	61.69
	SigLIP	98.95	87.05	87.35	74.80	87.04
	ResNet-50	98.35	86.10	89.25	64.95	84.66
AR	Histogram	64.95	35.85	71.15	39.55	52.88
	SigLIP	69.60	70.60	74.10	52.40	66.68
	ResNet-50	69.15	70.60	73.15	48.35	65.31

Table 4: Ablation on the feature extractor used by InfoShot on SynFlash under a fixed budget (0.5 fps). Metrics include frame-level Recall (R, %), Anomaly Retrieval (AR, %), and Temporal Ordering (TO, %). Best results are highlighted in bold , and second-best are underlined .

Figure 6: SynFlash evaluation results with different methods (same metrics as above) as function of fps, x-axis.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Case study of camouflage injection pattern.

Panel of Figure 8 — Figure 8: Case study of high-saturation injection pattern.

Panel of Figure 9 — Figure 9: Case study of curtain injection pattern.

Panel of Figure 10 — Figure 10: Case study of PiP injection pattern.

Explore similar work

May 21, 2026

Swift Sampling: Selecting Temporal Surprises via Taylor Series

While most frames in long-form video are redundant, the critical information resides in temporal surprises: moments where the actual visual features deviate from their predicted evolution. Inspired by the human brain's predictive coding, we introduce Swift Sampling, an elegant, training-free frame selection algorithm that automatically identifies high-information moments in a video. Specifically, we model a video as a differentiable trajectory in the visual latent space and compute the velocity and acceleration of its features. Then, we apply Taylor expansion to project the expected path of subsequent frames. Frames that diverge sharply from this predicted manifold are identified as temporally surprising frames and selected for sampling. Unlike prior training-free methods that rely on auxiliary networks or video-specific hyperparameter tuning, Swift Sampling is incredibly lightweight, adding only 0.02x additional computational cost over baseline making it 30x cheaper overhead than leading baselines. Across three long-video question answering benchmarks and 10 different downstream tasks, Swift Sampling outperforms uniform sampling and prior query-agnostic baselines. It is especially powerful for long videos with limited frame budgets improving accuracy by up to +12.5 points.

Dahye Kim, Bhuvan Sachdeva, Karan Uppal +3

Boston University · Microsoft Research India

Video Frame Selection Long-Video QA Temporal Video Understanding

Ask PDF

Mar 1, 2026

Event-Anchored Frame Selection for Effective Long-Video Understanding

Massive frame redundancy and limited context window make efficient frame selection crucial for long-video understanding with large vision-language models (LVLMs). Prevailing approaches, however, adopt a flat sampling paradigm which treats the video as an unstructured collection of frames. In this paper, we introduce Event-Anchored Frame Selection (EFS), a hierarchical, event-aware pipeline. Leveraging self-supervised DINO embeddings, EFS first partitions the video stream into visually homogeneous temporal segments, which serve as proxies for semantic events. Within each event, it then selects the most query-relevant frame as an anchor. These anchors act as structural priors that guide a global refinement stage using an adaptive Maximal Marginal Relevance (MMR) scheme. This pipeline ensures the final keyframe set jointly optimizes for event coverage, query relevance, and visual diversity. As a training-free, plug-and-play module, EFS can be seamlessly integrated into off-the-shelf LVLMs, yielding substantial gains on challenging video understanding benchmarks. Specifically, when applied to LLaVA-Video-7B, EFS improves accuracy by 4.7%, 4.9%, and 8.8% on VideoMME, LongVideoBench, and MLVU, respectively.

Wang Chen, Yongdong Luo, Yuhui Zeng +5

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · College of Computer and Data Science, Fuzhou University

Video Frame Selection Query-Aware Video Frame Selection Temporal Video Understanding Long-Context VLMs

Ask PDF

Jul 30, 2026

VisualRouter: Query-Grounded Visual Sampling for Long Video Understanding

Large vision-language models (LVLMs) have achieved significant progress in video understanding, yet understanding long videos remains challenging due to the large number of visual tokens and limited context windows. Visual sampling provides a practical solution by selecting an informative subset of frames. However, existing methods typically either rely on relevance-aware sampling, leading to redundant frame selection and insufficient temporal coverage, or adopt a fixed sampling strategy regardless of query type. In this paper, we propose VisualRouter, a training-free and plug-and-play framework for query-grounded visual sampling. VisualRouter first classifies each query as either global or local and then applies the corresponding sampling strategy. For global queries, it employs a relevance-coverage hybrid strategy that preserves temporal coverage while retaining query-relevant visual evidence. For local queries, it adopts an event-aware frame selection strategy that performs event partitioning, segment-level frame allocation, and intra-event frame selection, jointly balancing relevance, coverage, and diversity with a limited number of input frames. Experiments show that VisualRouter consistently improves multiple LVLMs over uniform sampling, achieving gains of 5.2%, 7.7%, and 11.6% on Video-MME, LongVideoBench, and MLVU with Qwen2.5-VL-7B, and outperforming existing training-free visual sampling methods under the same setting.

Haiyue Zhang, Yi Bin, Xun Jiang +5

Tongji University, Shanghai, China · University of Electronic Science and Technology of China, Chengdu, China

Large Vision-Language Models Long-Video Understanding Temporal Video Understanding Query-Aware Video Frame Selection

Ask PDF