EARS: Evidence-Aware Retrieval via Sound and Motion for Budgeted Video Reasoning

Shih-Yao Lin

cs.CVOct 26, 2026

EARS: Evidence-Aware Retrieval via Sound and Motion for Budgeted Video Reasoning

Authors: Shih-Yao Lin

Organizations: Independent Researcher

Abstract

Uniform frame sampling can miss brief evidence in sparse-event videos when the VLM sees only K frames. We propose EARS (Evidence-Aware Retrieval via Sound and motion), a training-free, query-free front-end that ranks frames using optical-flow motion and audio energy, then applies temporal non-maximum suppression for diversity. The evidence score alone has frame-level AP 0.39; used only to select frames for the VLM, EARS with temporal diversity reaches localization AP 0.94, compared with 0.88 under Uniform sampling. In the updated matched study, at K = 1 EARS reaches XD-subset F1 0.660 versus 0.614 for centered Uniform (the paired interval includes zero); earlier backbone studies use separate protocols. On mostly-silent UCF-Crime, however, EARS slightly trails Uniform. A single configuration is used across both benchmarks; the new controls do not repeat the original held-out search. EARS therefore targets low-budget videos whose salient evidence aligns with motion or audio; on dense narratives or stealthy events, uniform or semantic selection remains preferable, and cheap modality statistics cannot reliably route between them. Code: https://github.com/mikeslin-ir/EARS

Figures & tables

Figure 1: Evidence-aware frame selection for sparse-event video reasoning. Under

K{=}6

, (a) Uniform largely misses a brief critical event; (b) query-aware/learned selectors [ 31 , 26 , 23 ] recover it but require a query, training, or extra model passes; (c) EARS instead routes frames using training-free motion–audio evidence.

Figure 2: IC suppresses redundant temporal evidence. Vanilla top- $K$ clusters around one peak; a minimum gap $\tau$ retains diverse frames.

		UCF-Crime (440 entries)				XD subset (504 videos)
Method	$K$	Acc	P	R	F1	Acc	P	R	F1
Uniform	1	0.398	1.000	0.086	0.159	0.665	0.927	0.186	0.310
EARS	1	0.382	0.909	0.069	0.128	0.819	0.952	0.583	0.723
Uniform	2	0.459	0.919	0.197	0.324	0.730	0.886	0.382	0.534
EARS	2	0.432	0.870	0.162	0.273	0.891	0.946	0.775	0.852
Uniform	4	0.527	0.918	0.310	0.464	0.897	0.913	0.824	0.866

Table 1: Original-protocol classification: Qwen2-VL-7B [ 25 ] , binary prompt, greedy decoding. UCF includes duplicate entries. All rows are retained, not directly comparable to updated Sec. 4.5 . Best per cell is bold ; older multi-choice ablations use a separate protocol.

Figure 3: F1 vs frame budget

K

on UCF-Crime (left) and XD-Violence (right), plotted from the EARS and Uniform columns of Table 1 (original Qwen2-VL-7B protocol, not the updated centered-Uniform study). On XD-Violence the EARS curve sits above Uniform through

K\!=\!8

(F1

+0.41

K\!=\!1

+0.32

K\!=\!2

), crossing only at

K\!=\!10

. On mostly-silent UCF-Crime, Uniform remains modestly stronger at every budget. This contrast — durable gains when audio–motion evidence aligns with the event, negative transfer when it does not — is the empirical signature of EARS being an evidence router rather than a universal selector.

Panel of Figure 4 — Figure 4: Successful cases. Uniform frames (top), EARS frames (middle), and evidence with selection times (bottom). EARS captures the raised handgun and on-ice brawl that Uniform misses; characteristic failure modes are shown in the supplementary.

Panel of Figure 5 — Figure 5: Characteristic failures. Uniform (top), EARS (middle), and evidence (bottom). (a) An early peak clusters EARS picks and misses a later event. (b) In a silent clip, title-card motion distracts EARS from the subtle theft.

	XD-Violence			UCF-Crime
Method	$K{=}1$	$K{=}2$	$K{=}4$	$K{=}1$	$K{=}2$	$K{=}4$
Uniform	0.310	0.534	0.866	0.159	0.324	0.464
AKS [ 23 ] (q-aware)	0.861	0.895	0.890	0.242	0.346	0.412
SigLIP top- $K$ [ 34 ] (q-aware)	0.861	0.889	0.890	0.242	0.297	0.359
EARS (q-free)	0.723	0.852	0.912	0.128	0.273	0.419

Table 2: Query-aware selector baselines (F1 at tight budgets; same Qwen2-VL-7B [ 25 ] backbone, prompt, split, and frozen SigLIP relevance for all rows). The two query-aware selectors (AKS, SigLIP top-

K

) are strong when a task-specific anomaly query is available; EARS is the encoder-free, query-free option. Best per (dataset,

K

) in bold .

$K$	XD subset	UCF-Crime
1	$+0.017\ [-0.023,+0.057]$	$-0.196\ [-0.302,-0.093]$
2	$+0.022\ [-0.009,+0.054]$	$-0.028\ [-0.113,+0.058]$
4	$+0.042\ [+0.017,+0.067]$	$+0.009\ [-0.048,+0.066]$

Table 3: Hybrid minus semantic-NMS:

\Delta

F1 [95% interval]. Exploratory, unadjusted; 10,000 paired video resamples.

	$K=1$		$K=2$
Backbone	Unif.	EARS	Unif.	EARS
Qwen2.5-VL-7B [ 2 ]	.136	.561	.302	.656
Qwen3-VL-8B [ 1 ]	.433	.798	.599	.873
LLaVA-NeXT-7B [ 37 ]	.357	.693	.545	.788

Table 4: Backbone-transfer stress test on XD-Violence (F1). A selector configuration fixed across this transfer study improves every tested VLM at tight budgets under its original protocol, separate from the new controls.

Method	AP $\uparrow$	AUC $\uparrow$
$A_{t}$ proxy (no VLM)	0.393	0.583
Uniform + VLM	0.881	0.921
EARS + VLM (no IC)	0.925	0.938
EARS + IC + VLM	0.944	0.951

Table 5: Frame-level localization on XD-Violence. The evidence proxy is weak alone but effective for routing frames to the VLM.

Study	Setting	UCF	XD
Fusion	Audio only $(1,0)$	.291	.654
	Motion only $(0,1)$	.411	.593
	Balanced $(1,1)$	.394	.688
IC gap	$\tau=0$ s	.350	.778
	$\tau=0.5$ s	.427	.716
	$\tau=1$ s	.376	.671

Table 6: Component ablations (F1; matched multi-choice protocol). Fusion uses UCF

K\!=\!2

/XD

K\!=\!6

; IC uses

K\!=\!4

on both. Balanced fusion is robust across modalities, while the useful temporal gap depends on clip structure.

$\alpha$	$\beta$	$\tau$ (s)	XD	UCF
0	1	0.5	.706	.548
1	1	0	.749	.530
1	1	0.5	.769	.557
1	1	1	.768	.555

Table 7: Held-out sensitivity (mean F1 over

K=1,2

; 300 training videos/dataset). Both datasets independently select the same global configuration, and neighboring gaps are nearly tied.

	XD-Violence			UCF-Crime
Method	$K{=}1$	$K{=}2$	$K{=}4$	$K{=}1$	$K{=}2$	$K{=}4$
Uniform	.310	.534	.866	.159	.324	.464
Random	$.627_{\pm.024}$	$.767_{\pm.012}$	$.863_{\pm.007}$	$.148_{\pm.027}$	$.320_{\pm.024}$	$.414_{\pm.013}$
EARS	.723	.852	.912	.128	.273	.419

Table 8: Random-sampling control (F1; same frozen Qwen2-VL-7B protocol as Table 1 ). Random reports mean

\pm

std over five seeds; EARS and Uniform are deterministic.

Panel of Figure 14 — Figure S1 : Motion and audio evidence on real UCF-Crime videos. For each example we show normalized motion $M_{t}$ (orange), audio energy $E_{t}$ (purple), and fused evidence $A_{t}$ (green). The red band marks the ground-truth interval, gray dashed lines are Uniform picks, and green solid lines are EARS picks after IC. Below each plot are the six EARS-selected frames; green borders mark frames inside the event.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Selector	Per-task	Extra model / API	Repro.
(Sec. 2 )	query	in the loop	here
AKS [ 23 ]	text	SigLIP (frozen)	✓
SigLIP top- $K$ [ 34 ]	text	SigLIP (frozen)	✓
M-LLM-FS [ 12 ]	implicit	VLM per frame	partial
SeViLA [ 31 ]	QA	BLIP-2 loc. ( trained )	✗
VideoTree [ 27 ]	QA	LLM-guided CLIP tree	✗

Appendix

Table S1 : Selector requirements and matched evaluations. ✓ marks the two query-aware selectors evaluated under EARS’s exact backbone, prompt, and split (Table 2 ). Methods designed for question-conditioned long-video QA require a task query and, in some cases, trained or proprietary components; the concurrent 2026 methods were published after our evaluation was completed.

	XD-Violence			UCF-Crime
Method	$K{=}1$	$K{=}2$	$K{=}4$	$K{=}1$	$K{=}2$	$K{=}4$
Uniform (det.)	0.310	0.534	0.866	0.159	0.324	0.464
Random	$0.627_{\pm.024}$	$0.767_{\pm.012}$	$0.863_{\pm.007}$	$0.148_{\pm.027}$	$0.320_{\pm.024}$	$0.414_{\pm.013}$
EARS (det.)	0.723	0.852	0.912	0.128	0.273	0.419

Appendix

Table S2 : Random-7B seed variance (five seeds; same Qwen2-VL-7B backbone, binary prompt, and split as Table 1 ; mean

\pm

std F1). Uniform and EARS are deterministic and repeated from Table 1 for reference. Best per (dataset,

K

) in bold .

Method	Train-free	Selection signal	VLM passes / video	Modality	Year
AKS [ 23 ]	✓	CLIP–query sim.	1	V+T	’25
SeViLA [ 31 ]	✗	Localizer–VLM	$K\!+\!1$	V	’23
VideoAgent [ 26 ]	✓	Iterative agent loop	many	V+T	’24
M-LLM-FS [ 12 ]	✗	VLM scoring of frames	$\geq\!N/K$	V+T	’25
VideoTree [ 27 ]	✓	Hierarchical CLIP tree	$\log N$	V	’25
WFS-SB [ 7 ]	✓	Semantic boundaries	1	V+T	’26

Appendix

Table S3 : Frame-selection methods for video LLMs. Training-free semantic selectors have advanced rapidly, but remain query-conditioned and vision-centric; EARS is explicitly audio-visual and query-free.

N

= candidate frames; V/A/T denote visual, audio, text.

Method	$K\!=\!4$ Acc	$K\!=\!8$ Acc
Uniform	0.646	0.692
EARS (ours)	0.590	0.602
$\Delta$ (EARS $-$ Uniform)	$-0.056$	$-0.090$

Appendix

Table S4 : EgoSchema [ 20 ] MCQ accuracy on the 500-question subset (Qwen2.5-VL-7B-Instruct [ 2 ] , identical preprocessing). Bold marks the better method per column. EARS underperforms uniform sampling on this dense-narrative, audio-silent benchmark, consistent with its sparse-event design assumption.

		XD-Violence			UCF-Crime
Backbone	Sel.	$K{=}1$	$K{=}2$	$K{=}4$	$K{=}1$	$K{=}2$	$K{=}4$
Qwen2.5-VL-7B	Unif.	0.136	0.302	0.675	0.014	0.198	0.297
Qwen2.5-VL-7B	EARS	0.561	0.656	0.758	0.129	0.181	0.193
Qwen3-VL-8B	Unif.	0.433	0.599	0.873	0.151	0.325	0.418
Qwen3-VL-8B	EARS	0.798	0.873	0.906	0.351	0.381	0.397
LLaVA-NeXT-7B †	Unif.	0.357	0.545	0.764	—	—	—

Appendix

Table S5: Backbone transfer stress test. F1 (Uniform vs. EARS) for three additional VLMs under one selector configuration fixed across this transfer study; within this table only the VLM changes. The tight-budget gain transfers to newer Qwen generations and the cross-family LLaVA at

K\!\leq\!4

on XD. Gains bold where EARS

>

Uniform. † LLaVA-NeXT-Video-7B collapses to F1

0

K\!\geq\!8

on XD (its frame-count sweet spot is narrow) and is near the floor on UCF-Crime, which it cannot solve; we therefore read the cross-family evidence only where the backbone is competent (XD,

K\!\leq\!4

Method	AP $\uparrow$	AUC $\uparrow$
$A_{t}$ proxy alone (no VLM)	0.393	0.583
Uniform + VLM (baseline)	0.881	0.921
EARS, evidence + VLM (no IC)	0.925	0.938
EARS, evidence + IC + VLM	0.944	0.951

Appendix

Table S6 : Frame-level localization on XD-Violence. The raw evidence proxy

A_{t}

is a weak detector (AP

=

0.393) but, used as a router for the VLM, it lifts AP by

+6.3

over uniform sampling and a further

+1.9

when combined with Information Compression.

	UCF $K\!=\!2$		XD $K\!=\!6$
$(\alpha,\beta)$	Acc	F1	Acc	F1
$(1,0)$ audio-only	0.436	0.291	0.790	0.654
$(0,1)$ motion-only	0.491	0.411	0.758	0.593
$(1,1)$ balanced	0.482	0.394	0.802	0.688
$(2,1)$ audio-heavy	0.482	0.394	0.796	0.671
$(1,2)$ motion-heavy	0.484	0.398	0.786	0.652

Appendix

Table S7 : Ablations (matched multi-choice prompt under the same Qwen2-VL-7B backbone as Table 1 ; absolute F1 values are not directly comparable to Table 1 because the two tables use different prompts, but the relative rankings within each ablation are) . Left: modality fusion weights

(\alpha,\beta)

at the per-dataset budget. Motion dominates the silent-surveillance UCF set; balanced fusion dominates the multimodal XD set. Right: IC gap

\tau

K\!=\!4

. A small non-zero

\tau

is preferred on long, sparse UCF-Crime clips; vanilla top-

K

(

\tau\!=\!0

) is optimal on short XD-Violence clips.

Front-end	Encoder	Query	s/video
SigLIP relevance [ 34 ] (AKS, top- $K$ )	✓	✓	6.89
EARS , per-frame seeking	✗	✗	7.83
EARS , single-pass decode	✗	✗	3.12
Uniform + VLM, end to end	✗	✗	$1.33{\pm}0.53$
EARS + VLM, end to end	✗	✗	$6.87{\pm}2.95$
VLM-in-the-loop (SeViLA [ 31 ] , M-LLM-FS [ 12 ] )	✓	✓	$+\,\geq\!1$ VLM pass

Appendix

Table S8: Selection and end-to-end cost (24 XD-Violence videos, mean

299

frames/video). Frame decoding is shared by both front-ends and dominates unless it is done in a single pass; the SigLIP row still uses per-frame seeking, so the realistic gap is near

2\times

, not the

2.5\times

the front-end rows show. Paired end-to-end rows include Qwen2-VL-7B at

K\!=\!4

(mean

\pm

std).

Figure S2 : Adaptive-

K

Pareto on UCF-Crime (left) and XD-Violence (right). Sweeping

\theta

in Alg. 1 trades average frames-per-video against F1; crosses mark fixed-

K

Uniform at

K\!\in\!\{1,2,4,6,8,10\}

. UCF matches its best fixed-

\tau

<\!1/2

the frame cost; XD is flat.

Policy	Mean score
Per-video router (modality statistics)	0.635
Always Uniform	0.647
Budget rule (EARS iff $K\!\leq\!\theta$ )	0.676
Always EARS	0.685
Per-video oracle (upper bound)	0.789

Appendix

Table S9: Routing policies , mean over ten (dataset,

K

) cells (F1 for UCF/XD, accuracy for EgoSchema). The oracle shows substantial headroom, but the learned per-video router underperforms both fixed policies. Thresholds for both adaptive rows are selected leave-one-dataset-out.

Figure S3 : Precision–Recall (left) and ROC (right) curves on the frame-level XD-Violence annotations.

Study	Quantitative finding
Flow estimator	TV-L1 [ 32 ] stays within $0.024$ F1 of Farneback [ 9 ] for XD at every $K\leq 4$ .
Dense narrative	EgoSchema [ 20 ] EARS $-$ Uniform accuracy is $-0.056/-0.090$ at $K=4/8$ .
Per-video routing	Modality-statistics router $0.635$ ; always-EARS $0.685$ ; oracle upper bound $0.789$ .
End-to-end cost	Paired $n\!=\!24$ : EARS $6.87{\pm}2.95$ vs. Uniform $1.33{\pm}0.53$ s/video; reusable CPU preprocessing dominates.

Appendix

Table S10 : Original robustness, scope, and efficiency summary, relocated from the main paper without changing its numerical findings. Values are F1 unless noted; latency is measured before cache reuse.

Query-independent selection; no learned selection encoder
	XD subset ( $n=504$ )			UCF-Crime ( $n=290$ )
Selector	$K=1$	$K=2$	$K=4$	$K=1$	$K=2$	$K=4$
Uniform	0.614	0.791	0.893	0.303	0.528	0.713
Random (3 seeds)	$0.603\pm 0.010$	$0.779\pm 0.005$	$0.853\pm 0.015$	$0.249\pm 0.030$	$0.537\pm 0.028$	$0.668\pm 0.019$
Motion	0.524	0.743	0.856	0.270	0.471	0.685
Audio	0.683	0.838	0.907	0.095	0.158	0.272

Appendix

Table S11: Updated matched F1, all methods and budgets. Random is mean

\pm

sample SD over three streams. The upper/lower groups distinguish selection without/with a learned semantic encoder and task query.

Selector	$K$	Acc.	Prec.	Recall	F1	TP	FP	FN	TN
Uniform	1	0.768	0.939	0.456	0.614	93	6	111	294
Uniform	2	0.851	0.916	0.696	0.791	142	13	62	287
Uniform	4	0.917	0.931	0.858	0.893	175	13	29	287
EARS	1	0.786	0.921	0.515	0.660	105	9	99	291
EARS	2	0.867	0.920	0.735	0.817	150	13	54	287
EARS	4	0.919	0.914	0.882	0.898	180	17	24	283

Appendix

Table S12: Updated XD subset: complete primary metrics and confusion counts, including all three Random streams. Every row covers 504 videos.

Selector	$K$	Acc.	Prec.	Recall	F1	TP	FP	FN	TN
Uniform	1	0.603	1.000	0.179	0.303	25	0	115	150
Uniform	2	0.679	0.912	0.371	0.528	52	5	88	145
Uniform	4	0.772	0.911	0.586	0.713	82	8	58	142
EARS	1	0.583	0.952	0.143	0.248	20	1	120	149
EARS	2	0.659	0.936	0.314	0.471	44	3	96	147
EARS	4	0.741	0.922	0.507	0.654	71	6	69	144

Appendix

Table S13: Updated UCF: complete primary metrics and confusion counts. Every row covers 290 unique videos with annotation-derived labels.

Cohort	Selector	$K$	Acc.	Prec.	Recall	F1	TP	FP	FN	TN
XD subset	Semantic top- $K$	1	0.857	0.824	0.824	0.824	168	36	36	264
XD subset	Semantic top- $K$	2	0.865	0.796	0.897	0.843	183	47	21	253
XD subset	Semantic top- $K$	4	0.847	0.751	0.931	0.832	190	63	14	237
XD subset	Semantic + NMS	1	0.857	0.824	0.824	0.824	168	36	36	264
XD subset	Semantic + NMS	2	0.865	0.788	0.912	0.845	186	50	18	250
XD subset	Semantic + NMS	4	0.859	0.755	0.966	0.847	197	64	7	236

Appendix

Table S14 : Updated semantic and targeted-control metrics. Same cohorts, reasoner, binary prompts, and budgets as the primary controls.

Cohort	$K$	$\Delta$ F1	95% paired interval	Wins	Losses	Ties
XD subset	1	$+0.047$	$[-0.038,\ +0.132]$	58	49	397
XD subset	2	$+0.026$	$[-0.030,\ +0.085]$	50	42	412
XD subset	4	$+0.005$	$[-0.033,\ +0.044]$	30	29	445
UCF-Crime	1	$-0.055$	$[-0.166,\ +0.060]$	13	19	258
UCF-Crime	2	$-0.057$	$[-0.168,\ +0.055]$	27	33	230
UCF-Crime	4	$-0.059$	$[-0.134,\ +0.018]$	20	29	241

Appendix

Table S15: Updated EARS-minus-Uniform paired comparisons, all budgets. Intervals use 10,000 joint video resamples, seed 42, and are unadjusted.

Cohort	Selector	Reference	$K$	$\Delta$ F1	95% interval
UCF-Crime	Semantic + NMS	Semantic top- $K$	1	$+0.000$	$[+0.000,+0.000]$
UCF-Crime	Semantic + NMS	Semantic top- $K$	2	$+0.028$	$[-0.003,+0.061]$
UCF-Crime	Semantic + NMS	Semantic top- $K$	4	$+0.037$	$[+0.009,+0.069]$
UCF-Crime	Semantic + EARS	Semantic + NMS	1	$-0.196$	$[-0.302,-0.093]$
UCF-Crime	Semantic + EARS	Semantic + NMS	2	$-0.028$	$[-0.113,+0.058]$
UCF-Crime	Semantic + EARS	Semantic + NMS	4	$+0.009$	$[-0.048,+0.066]$

Appendix

Table S16: All 36 updated targeted-control contrasts, including frame differences versus EARS, NMS-only effects, and activity fusion. Exploratory unadjusted 95% paired intervals, not confirmatory tests.

Cohort	Selector	Features	Selection	RGB	VLM	Total
XD subset	Uniform	Fresh	0.0002	0.2281	0.5215	0.7657
UCF-Crime	Uniform	Fresh	0.0002	0.1314	0.4664	0.6009
XD subset	Uniform	Cached	0.0002	0.2244	0.5222	0.7623
UCF-Crime	Uniform	Cached	0.0002	0.1318	0.4707	0.5973
XD subset	EARS	Fresh	0.7656	0.0971	0.5262	1.3775
UCF-Crime	EARS	Fresh	0.3958	0.1263	0.4685	0.9773

Appendix

Table S17: Updated cost stages in seconds: medians over per-video repetition medians. Fresh/cached refer to video features, not a cold/warm model or OS file cache.

Cohort	Selector	Features	$\Delta$ seconds	95% interval	Ratio
UCF-Crime	EARS	Fresh	$+0.3856$	$[+0.2739,+0.6267]$	1.652
UCF-Crime	EARS	Cached	$-0.0079$	$[-0.0146,+0.0131]$	0.987
UCF-Crime	Semantic top- $K$	Fresh	$+0.6239$	$[+0.5952,+0.7634]$	2.084
UCF-Crime	Semantic top- $K$	Cached	$-0.0255$	$[-0.1512,-0.0030]$	0.956
XD subset	EARS	Fresh	$+0.6470$	$[+0.5045,+1.3221]$	1.967
XD subset	EARS	Cached	$-0.0278$	$[-0.1079,+0.0071]$	0.954

Appendix

Table S18: All updated paired total-cost comparisons with Uniform. Video-bootstrap intervals and median paired total ratios; positive differences or ratios above one indicate greater cost.

Video Frame Selection

Video frame selection chooses a compact set of frames that preserves important events, context, or task-specific evidence while reducing processing costs.

See all 65 papers

One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding
Aug 6, 2026Wang Chen, Yu Chen, Xiang Wang +3Long-Video Understanding Video Frame Selection
Media Analytics and Computing Lab, Xiamen University, Xiamen, China · Amap, Alibaba Group, China · Department of Computer Science, University of Rochester, Rochester, NY, USA
When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding
Aug 4, 2026Ke Li, Jiayu Chen, Maoliang Li +5Efficient VLM Inference Large Vision-Language Models
School of Electronics Engineering and Computer Science, Peking University · School of Computer Science, Peking University
RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding
Aug 30, 2026Shanqing Xu, Meng Luo, Mengchen Qian +7Large Vision-Language Models Long-Video Understanding
Huazhong University of Science and Technology · National University of Singapore · Fudan University

Aug 6, 2026J/K move · Enter open · S save

One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding

Wang Chen, Yu Chen, Xiang Wang, Shuai Li+2

Media Analytics and Computing Lab, Xiamen University, Xiamen, China · Amap, Alibaba Group, China · Department of Computer Science, University of Rochester, Rochester, NY, USA

Long-Video Understanding Video Frame Selection Query-Aware Video Frame Selection Adaptive Video Frame Sampling

Open paper AskPDF arXiv

Frame selection is essential for applying Large Multimodal Models (LMMs) to long videos due to severe frame redundancy and limited context windows. Since the appropriate frame budget varies with the downstream LMM, reasoning demands, and latency constraints, a practical selector should serve multiple budgets. However, existing methods typically optimize an isolated frame subset for each predefined budget: when the budget changes, previously selected evidence may be replaced rather than progressively augmented. A fixed-weight ranking allows prefix reuse across budgets but applies the same weighting at every position, overlooking the distinct roles of early and later ranks. We formulate long-video frame selection as a Matryoshka ranking problem: constructing a single priority sequence whose small prefixes concentrate query-conditioned evidence, while progressively larger prefixes preserve this evidence and add broader temporal context. Efficiently constructing such a ranking is itself challenging, as densely sampling long videos and evaluating frame-query relevance incurs substantial overhead. We therefore introduce Matryoshka Evidence-to-Context (MEC) Frame Selection, a training-free framework that builds a reusable sparse video index, discovers candidates through sparse probing and local zooming, and greedily constructs a position-adaptive ranking - early positions emphasize evidence; later positions progressively favor temporal coverage while preserving visual diversity. A single ranking can thus be truncated to any target budget without rerunning the selector. Across four benchmarks and six frame budgets, MEC improves average accuracy over uniform sampling by 3.77 points, matches strong state-of-the-art selectors, and reduces end-to-end selection latency by 47.37-51.19%.

Abstract

Video Frame Selection

Explore similar work

One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding

When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding

RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding