dKFD: Phase-Structured Evidence Allocation for Fixed-Budget Localized Event Understanding

Aditya Bagri; Ashutosh Kumar; Chaitanya Lakhchaura; Avinash Anand; Zhengkui Wang; Rajiv Ratn Shah

cs.CVSep 27, 2026

dKFD: Phase-Structured Evidence Allocation for Fixed-Budget Localized Event Understanding

Authors: Aditya Bagri, Ashutosh Kumar, Chaitanya Lakhchaura, Avinash Anand, Zhengkui Wang, Rajiv Ratn Shah

Organizations: Indraprastha Instituition of Information Technology, New Delhi, India · Rochester Institute of Technology, NY, USA · SIT × Nvidia AI Center (SNAIC), Singapore · Indian Institute of Technology Kanpur, Kanpur, India

Abstract

Sparse video understanding often requires selecting a small set of visual evidence under a fixed frame budget. Most sparse selectors allocate this budget globally, allowing all frames to compete with one another. For temporally localized events, this can be a poor inductive bias: useful evidence is often distributed across pre-event context, the event itself, and post-event consequences. We study fixed-budget evidence allocation for localized event videos and show that globally competitive Top- $K$ selectors can preserve recognition and grounding while producing unstable event evidence. On DoTA Video Anomaly Recognition, Global Top- $K$ obtains competitive recognition and temporal grounding, but low selector-event alignment at $K=12$ (Frame AUC $51.5 \pm 10.8$ ). We propose dKFD, a phase-structured differentiable selector that reserves evidence capacity across pre-event, event, and post-event phases after full-sequence temporal encoding. Under matched-budget multi-seed evaluation, dKFD improves Frame AUC by $+30.97$ over a matched Global Top- $K$ selector at $K=12$ ( $p<0.01$ ), while yielding modest but statistically significant recognition gains and comparable temporal grounding. Mechanism ablations show that phase supervision is load-bearing: removing it reduces Frame AUC to $41.1 \pm 12.0$ even when phase-partitioned budgets are retained. Downstream diagnostics on VRU-Accident show consistent gains over learned Global Top- $K$ across VLM families, while dense captioning reveals a boundary condition where uniform sampling remains competitive. These results support phase-structured allocation as a controlled fixed-budget approach for event-centric sparse evidence selection, not as a universal video summarization strategy.

Figures & tables

Setting	Selection objective	What “good” selection means	What this setting does not directly test
Video summarization	Preserve broad video content	Coverage, diversity, representativeness	Whether supervised phase allocation produces event-aligned evidence under fixed $K$
Temporal grounding / action detection	Predict event interval	Accurate temporal boundaries	Whether sparse, exportable evidence is preserved alongside localization
Long-video VLM compression	Reduce visual tokens for LLM reasoning	Efficient answer generation under token limits	Whether explicit visual-frame allocation under fixed $K$ benefits from phase structure
dKFD setting	Allocate fixed-budget evidence	Supervised pre/event/post evidence under fixed $K$	This paper

Table 1: Positioning of dKFD relative to neighboring video-selection and long-video understanding settings. dKFD studies supervised fixed-budget evidence allocation for temporally localized events.

Figure 1: dKFD overview. The full video is temporally encoded before phase-aware selectors allocate a fixed evidence budget across pre-event, event, and post-event frames. The selected evidence pack is used for classification, while selector maps and pooled evidence condition temporal grounding over the full timeline.

Method	AVG Top-1 $\uparrow$	Macro-F1 $\uparrow$	tIoU $\uparrow$	Time Err. $\downarrow$	Frame AUC $\uparrow$
Global Top- $K$	$30.46\pm 0.42$	$28.25\pm 0.85$	$53.22\pm 0.62$	$9.41\pm 0.21$	$51.48\pm 10.81$
dKFD	$32.10^{**}\pm 0.75$	$30.48^{**}\pm 0.52$	$53.69\pm 0.45$	$9.18\pm 0.21$	$82.45^{**}\pm 0.65$
$\Delta$	$+1.64$	$+2.23$	$+0.47$	$-0.23$	$+30.97$

Table 2: Main controlled DoTA comparison at

K=12

over five seeds. Asterisks indicate paired seed-level significance against Global Top-

K

{}^{**}p<0.01

{}^{***}p<0.001

Setting	AVG Top-1 $\uparrow$	Macro-F1 $\uparrow$	tIoU $\uparrow$	Frame AUC $\uparrow$
Global, $K=8$	$30.34\pm 1.47$	$28.75\pm 1.42$	$52.96\pm 0.80$	$54.17\pm 13.79$
dKFD, $K=8$	$31.84\pm 1.88$	$30.11\pm 1.40$	$53.24\pm 0.38$	$81.39^{*}\pm 0.62$
Global, $K=12$	$30.46\pm 0.42$	$28.25\pm 0.85$	$53.22\pm 0.62$	$51.48\pm 10.81$
dKFD, $K=12$	$32.10^{**}\pm 0.75$	$30.48^{**}\pm 0.52$	$53.69\pm 0.45$	$82.45^{**}\pm 0.65$
Global, $K=16$	$30.33\pm 1.35$	$27.99\pm 0.57$	$53.54\pm 0.65$	$50.39\pm 8.02$
dKFD, $K=16$	$31.51\pm 1.40$	$29.72\pm 1.30$	$53.75\pm 0.52$	$82.47^{***}\pm 0.85$

Table 3: Budget scaling under matched frozen features and five-seed evaluation. Asterisks indicate paired significance against matched Global Top-

K

at the same

K

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

Figure 2: Evidence-quality summary on DoTA. (a) Frame AUC versus evidence budget

K

: dKFD stays tightly clustered near

82\%

across budgets, whereas Global Top-

K

shows a wide per-seed spread (shaded

\pm

std) straddling chance — capturing both budget-robustness and seed instability in a single view. (b) Hard-frame overlap by temporal region at

K=12

: dKFD’s gain over Global Top-

K

comes from the pre- and post-event regions, not from over-selecting the event window. (c) Phase-supervision sweep: removing phase supervision (

\lambda_{\mathrm{phase}}=0

) collapses both Frame AUC and phase coverage, while moderate supervision recovers stable alignment — phase supervision is load-bearing.

Variant	AVG Top-1 $\uparrow$	Macro-F1 $\uparrow$	tIoU $\uparrow$	Time Err. $\downarrow$	Frame AUC $\uparrow$
Full dKFD	$31.94\pm 0.23$	$30.28\pm 0.44$	$53.69\pm 0.45$	$9.11\pm 0.18$	$82.16\pm 0.69$
w/o temporal encoder	$32.32\pm 1.06$	$30.54\pm 0.89$	$39.86\pm 0.92$	$14.32\pm 0.24$	$66.00\pm 0.60$
w/o phase heads	$30.17\pm 1.34$	$28.28\pm 1.42$	$53.63\pm 0.42$	$9.35\pm 0.07$	$48.66\pm 9.71$
w/o selector-aware grounding	$30.80\pm 1.40$	$29.24\pm 1.22$	$52.98\pm 0.56$	$9.35\pm 0.18$	$82.70\pm 0.57$
w/o evidence summary	$31.17\pm 1.09$	$29.53\pm 0.87$	$53.30\pm 0.96$	$9.24\pm 0.16$	$82.41\pm 0.14$
w/o selector maps	$32.17\pm 1.31$	$30.21\pm 0.95$	$53.55\pm 0.40$	$9.15\pm 0.07$	$81.93\pm 0.46$

Table 4: Mechanism ablations on DoTA over five seeds. Temporal encoding mainly supports grounding, while phase-aware heads mainly support selector–event alignment.

$\lambda_{\mathrm{phase}}$	AVG Top-1 $\uparrow$	Macro-F1 $\uparrow$	tIoU $\uparrow$	Frame AUC $\uparrow$	Phase cov. $\uparrow$
$0$	$30.78\pm 2.25$	$28.98\pm 1.34$	$54.40\pm 1.01$	$41.12\pm 11.99$	$16.3\pm 15.3$
$0.05$	$30.17\pm 0.95$	$28.71\pm 0.29$	$53.60\pm 0.64$	$81.34\pm 0.94$	$80.2\pm 0.9$
$0.1$	$30.34\pm 0.64$	$28.79\pm 0.40$	$53.34\pm 1.27$	$82.02\pm 0.66$	$80.0\pm 1.4$
$0.2$	$31.95\pm 0.56$	$30.26\pm 0.44$	$53.42\pm 0.53$	$81.88\pm 0.67$	$80.6\pm 1.0$
$0.3$	$30.97\pm 1.00$	$29.42\pm 1.29$	$53.65\pm 0.37$	$82.59\pm 0.67$	$80.1\pm 1.4$
$0.5$	$30.88\pm 1.01$	$29.41\pm 1.17$	$54.45\pm 0.23$	$82.78\pm 0.86$	$80.3\pm 0.6$

Table 5: Sensitivity to phase supervision over three seeds. Without phase supervision, Frame AUC and phase coverage drop sharply despite retaining phase-partitioned budgets.

Figure 3: Qualitative selector comparison at $K=12$ on two DoTA test clips (left: OC, oncoming; right: VP, vulnerable road user). The cream band marks the annotated event interval; the three lanes per clip show Uniform sampling, Global Top- $K$ , and dKFD. Uniform covers the timeline but is not task-supervised; Global Top- $K$ concentrates frames near the event core; dKFD allocates the same budget across pre-event, event, and post-event phases, covering causal setup, interaction, and aftermath.

Input	WL	TE	RC	AT	AC	PM	AVG $\uparrow$
Full video	$63.80$	$72.10$	$39.40$	$47.30$	$25.60$	$44.70$	$48.82$
Global Top- $K$	$66.90$	$81.40$	$38.20$	$51.20$	$26.80$	$47.00$	$51.92$
Uniform	$69.60$	$82.90$	$36.30$	$52.30$	$28.60$	$46.90$	$52.77$
dKFD	$67.90$	$82.60$	$38.40$	$53.00$	$28.20$	$47.30$	$52.90$

Table 6: VRU-Accident VQA on Qwen2.5-VL-3B at

K=12

using 6,000 questions and the official evaluator. Extended multi-VLM results are provided in the supplementary material.

Figure 4: Dense-captioning use case on a VRU-Accident clip (“pedestrian runs out from behind an overtaken vehicle”). The $K{=}12$ dKFD evidence pack shown above (frame borders mark pre/event/post phases) captures the pedestrian’s emergence, the crossing, the impact, and the aftermath; feeding only these frames to Qwen2.5-VL produces a coherent collision narrative. Predicted (dKFD $\to$ VLM): “The video captures a collision between a car and a pedestrian on a city street… a white car is seen approaching… The pedestrian, dressed in a white shirt and dark pants, is crossing the street in front of the car… As the car approaches, it makes a right turn onto the next lane… the pedestrian’s path intersects with the car’s path, leading to a collision.” Ground truth (expert, abridged): “On an overcast afternoon… a female pedestrian… suddenly runs from the right sidewalk into the roadway at a diagonal angle… The driver’s view of the pedestrian is obstructed by the white vehicle… the driver fails to brake in time, resulting in a collision… The vehicle continues traveling straight without any apparent evasive action…”

Explore similar work

RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding
Aug 30, 2026Shanqing Xu, Meng Luo, Mengchen Qian +7Large Vision-Language Models Long-Video Understanding
Huazhong University of Science and Technology · National University of Singapore · Fudan University
One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding
Aug 6, 2026Wang Chen, Yu Chen, Xiang Wang +3Long-Video Understanding Video Frame Selection
Media Analytics and Computing Lab, Xiamen University, Xiamen, China · Amap, Alibaba Group, China · Department of Computer Science, University of Rochester, Rochester, NY, USA
CREST: Curvature-Regulated Event-Centric Sampling for Efficient Long-Video Understanding
May 9, 2026Mehrajul Abadin Miraj, Abdul Mohaimen Al Radi, Shariful Islam Rayhan +4Data Selection Long-Video Understanding
Dept. of CSE, University of Dhaka · Dept. of CSE, University of Central Florida

Aug 30, 2026J/K move · Enter open · S save

RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding

Shanqing Xu, Meng Luo, Mengchen Qian, Yuhui Gao+6

Huazhong University of Science and Technology · National University of Singapore · Fudan University

Large Vision-Language Models Long-Video Understanding Video Frame Selection Query-Aware Video Frame Selection

Open paper AskPDF arXiv

Long videos contain far more visual content than Large Vision-Language Models (LVLMs) can process under a fixed visual-token budget, making frame selection essential. Existing query-aware selectors usually estimate frame-query relevance and build a compact subset from high-scoring frames. Although their mechanisms differ, the similarity sequence is still often treated primarily as values to rank or sample from, rather than as an ordered signal whose shape reflects how query-relevant evidence emerges, peaks, and fades over time. This can obscure frames that explain, contextualize, or follow an event, because such evidence may lie on the rising or falling sides of a nearby relevance peak and receive lower absolute scores. We propose RIDGE, a frame selection framework that reads the frame-query similarity curve as a temporal signal. By using local changes and curvature, RIDGE partitions the timeline into structural regions and applies region-specific selection to preserve event cores, transitions, buildup, aftermath, and contextual frames under a fixed budget. It is a lightweight post-processing step on precomputed frame-query scores and requires neither training nor iterative LVLM calls. Across four long-video benchmarks and three backbones, RIDGE achieves the best performance in most settings and remains competitive in the others.

Abstract

Explore similar work

RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding

One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding

CREST: Curvature-Regulated Event-Centric Sampling for Efficient Long-Video Understanding