Video frame sampling is essential for efficient long-video understanding with Vision-Language Models (VLMs), since dense inputs are costly and often exceed context limits. Yet when only a small number of frames can be retained, existing samplers often fail to balance broad video coverage with brief but critical events, which can lead to unreliable downstream predictions. To address this issue, we present InfoShot, a task-agnostic, shot-aware frame sampler for long-video understanding. InfoShot first partitions a video into semantically consistent shots, and then selects two complementary keyframes from each shot: one to represent the main content and one to capture unusual within-shot changes. This design is guided by an information-theoretic objective that encourages the sampled set to retain high information about both shot structure and sparse within-shot deviations. In this way, it improves the chance of preserving both overall video context and short decision-critical moments without requiring any retraining. To better evaluate such short-lived events, we further introduce SynFlash, a synthetic benchmark with controllable sub-second anomaly patterns and frame-level ground truth, and we also evaluate InfoShot on existing anomaly datasets and general video understanding tasks. Experiments show that InfoShot improves anomaly hit rate and downstream Video-QA accuracy under frame number constraints, while matching or outperforming strong baselines on standard video understanding benchmarks.
Figures & tables
Figure 1: The overall framework of our approach. We propose a task-agnostic and plug-and-play sampler, InfoShot, before the VLM to select keyframes under a fixed budget. InfoShot performs greedy shot segmentation in deep feature space (Sec. 3.4 ) and then selects two frames per shot: a common and a unique high-deviation frames (Sec. 3.5 ). The downstream VLM uses the same prompt; only the sampled frame set differs.
Figure 2: Overview of the SynFlash generation pipeline. ImageNet-1k [ 27 ] images are injected at random times into videos formed by concatenating three Panda-70M [ 7 ] clips, creating four transient anomaly types with frame-level ground truth.
Figure 3: Case study from SynFlash (PiP). Left: sampled frames from different methods. Right: example questions.
Metric
Method
Camou.
High-Sat.
Curtain
PiP
Avg.
R
Uniform
11.20
10.00
19.15
11.20
11.81
VSUMM
74.50
9.15
93.75
9.35
46.54
TransNet
95.55
16.55
81.15
10.30
46.28
InfoShot
98.35
87.05
88.45
64.95
84.70
AR
Uniform
29.60
29.05
34.70
29.75
30.78
VSUMM
56.75
28.40
74.95
29.45
47.39
Table 1: Performance comparison on SynFlash. Metrics include frame-level Recall (R, %), Anomaly Retrieval Accuracy (AR, %), and Temporal Ordering Accuracy (TO, %). Best results are highlighted in bold , and second-best are underlined .
Figure 4: Feature-space distortion Dist(K) (lower is better) on SynFlash.
Method
UCF-Crime
XD-Violence
Weighted Avg.
ER
CR
ER
CR
ER
CR
TransNet
40.97
37.50
37.16
39.00
37.56
38.69
VSUMM
65.97
62.50
57.35
51.60
58.25
53.82
Uniform
69.44
65.62
59.53
54.20
60.56
56.53
InfoShot
69.44
66.41
63.09
58.40
63.75
60.03
Table 2: Event coverage on HIVAU-70k annotations under 0.1 fps. ER: event recall, CR: complete coverage rate. Weighted Avg. is weighted by the number of events/videos in each dataset. Best results are highlighted in bold , and second-best are underlined .
Method
Short
Med.
Avg.
TransNet
65.8
60.4
63.2
VSUMM
67.8
62.5
65.2
Uniform
71.8
62.2
67.0
InfoShot
71.5
62.4
67.0
Table 3: Video-MME results under the same frame budget. Best results are in bold , and second-best are underlined
Figure 5: Similarity matrices on SynFlash under different feature extractors. The first row uses HSV color-histogram features and the second row uses semantic features. Each column shows one example video from a SynFlash subset (Camouflage, High-Sat, Curtain, and PiP). Red arrows indicate the temporal injected flash event.
Metric
Method
Camou.
High-Sat.
Curtain
PiP
Avg.
R
Histogram
93.65
27.05
85.10
40.95
61.69
SigLIP
98.95
87.05
87.35
74.80
87.04
ResNet-50
98.35
86.10
89.25
64.95
84.66
AR
Histogram
64.95
35.85
71.15
39.55
52.88
SigLIP
69.60
70.60
74.10
52.40
66.68
ResNet-50
69.15
70.60
73.15
48.35
65.31
Table 4: Ablation on the feature extractor used by InfoShot on SynFlash under a fixed budget (0.5 fps). Metrics include frame-level Recall (R, %), Anomaly Retrieval (AR, %), and Temporal Ordering (TO, %). Best results are highlighted in bold , and second-best are underlined .
Figure 6: SynFlash evaluation results with different methods (same metrics as above) as function of fps, x-axis.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Case study of camouflage injection pattern.
Figure 8: Case study of high-saturation injection pattern.
Figure 9: Case study of curtain injection pattern.
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · College of Computer and Data Science, Fuzhou University