CoRE: Weakly Supervised Coarse-to-Fine Risk Evidence Learning in Driving Videos

Kaiser Hamid; Can Cui; Nade Liang

cs.CVAug 26, 2026

CoRE: Weakly Supervised Coarse-to-Fine Risk Evidence Learning in Driving Videos

Authors: Kaiser Hamid, Can Cui, Nade Liang

Organizations: Texas Tech University · Purdue University

Abstract

Perceived risk in driving evolves over time and may be supported by specific scene entities, yet supervision is typically limited to coarse video-level judgments. Learning when supporting evidence emerges and which entities support a risk predictor would ordinarily require costly temporal- and entity-level annotations. We introduce CoRE, a weakly supervised coarse-to-fine framework that learns fine-grained prediction support from coarse video supervision. CoRE first trains a video-level predictor and then freezes it. Structured interventions over candidate temporal regions or entity tracks measure how each candidate changes the coarse prediction, producing graded prediction-effect targets. These targets are distilled into a student that directly predicts temporal and entity support from the original video, without requiring interventions at inference. We evaluate this learning principle across three complementary settings: RISEE tests perceived-risk support from subjective clip-level judgments without temporal or entity-level risk annotations; DoTA provides independent temporal event annotations for evaluating weakly supervised traffic-anomaly localization; and UCF-Crime tests whether the same coarse-to-fine mechanism extends to a standard non-driving anomaly-detection benchmark. Across these settings, CoRE learns informative fine-grained support from coarse supervision, with strong temporal localization on DoTA and competitive performance on UCF-Crime. These results show that coarse video predictions can provide useful supervision for recovering the fine-grained evidence supporting them, without requiring corresponding fine-grained labels.

Figures & tables

Figure 1: CoRE decomposes coarse video-level perceived risk into fine-grained support, revealing how risk evolves over time, what visual evidence drives the prediction, and how strongly individual objects contribute.

Panel of Figure 2 — Figure 2: Overview of CoRE. A coarse video predictor is trained from video-level supervision and frozen. Structured interventions over temporal and entity candidates produce prediction effects that are converted into graded support targets and distilled into a student. At inference, the student directly predicts the coarse response together with temporal and entity support in a single forward pass.

Method	MAE $\downarrow$	RMSE $\downarrow$	Spearman $\rho$ $\uparrow$	PairAcc $\uparrow$
Frame-average regression	0.640	0.788	0.581	0.721
Attention MIL [ 14 ]	0.577	0.709	0.653	0.750
Soft top- $k$ MIL [ 34 ]	0.683	0.815	0.575	0.713
Temporal convolution	0.649	0.808	0.572	0.710
Temporal Transformer	0.659	0.801	0.552	0.703
Video Swin regression [ 19 ]	0.592	0.718	0.641	0.742

Table 1: Clip-level perceived-risk prediction on RISEE. Scenario-level five-fold evaluation using only human clip-level ratings. Best per metric in bold , second best underlined .

Method	Sel. Drop $\uparrow$	Gain/Rand. $\uparrow$	Effect $\rho$ $\uparrow$	Top Perc. $\uparrow$	Pos. Rate $\uparrow$
Uniform weighting	0.045	-0.049	0.000	0.538	0.726
Attention MIL [ 14 ]	0.032	-0.050	-0.167	0.482	0.620
Soft top- $k$ MIL [ 34 ]	0.050	-0.038	-0.214	0.474	0.592
CoRE (ours)	0.320	0.245	0.542	0.888	0.989

Table 2: Temporal prediction support on RISEE. Held-out interventions measure prediction dependence; Effect

\rho

measures agreement with candidate effects, while Top Perc. and Pos. Rate summarize selection quality.

Features	Method	Venue	Frame AUC $\uparrow$	Frame AP $\uparrow$	Macro AUC $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	Best tIoU $\uparrow$
ResNet-50	Top- $k$ MIL §		0.377	0.251	0.381	0.499	0.147	0.327
	Deep MIL † [ 34 ]	CVPR’18	0.520	0.335	0.526	0.462	0.150	0.314
	RTFM † [ 37 ]	ICCV’21	0.565	0.349	0.569	0.543	0.191	0.352
	MGFN † [ 5 ]	AAAI’23	0.642	0.407	0.635	0.493	0.149	0.322
	UR-DMU † [ 45 ]	AAAI’23	0.574	0.398	0.582	0.488	0.143	0.316
	OE-CTST ‡ [ 23 ]	WACV’24	0.387	0.257	0.396	0.349	0.139	0.361

Table 3: Weakly supervised temporal anomaly localization on DoTA. All results use the same WS-DoTA split and evaluation pipeline. Methods within each block share the same visual feature bank. † denotes a protocol-matched adaptation based on released official code; ‡ denotes our reimplementation from the paper/supplement when executable official code was unavailable; § denotes our controlled baseline. Best in bold ; second best underlined .

Panel of Figure 3 — Figure 3: Qualitative temporal localization on UCF-Crime. Each example shows video frames, the frame-level anomaly interval (pink), and the temporal scores produced by CoRE and MGFN [ 5 ] . Red borders identify displayed frames within the annotated interval.

Panel of Figure 4 — Figure 4: Qualitative temporal localization on DoTA. Ground-truth intervals are compared with OE-CTST [ 23 ] , PE-MIL [ 4 ] , and CoRE on the same videos.

Panel of Figure 5 — Figure 5: Temporal support on contrasting RISEE clips. CoRE rises with perceived risk and remains suppressed in the low-risk clip; $t_{1}$ – $t_{5}$ mark the displayed frames. Attn. MIL ; soft top- $k$ ; CoRE .

Method	Venue	Feature	AUC (%) $\uparrow$
MIL-Rank [ 34 ]	CVPR’18	C3D RGB	75.41
MIST [ 8 ]	CVPR’21	I3D RGB	82.30
RTFM † [ 37 ]	ICCV’21	I3D RGB	84.30
NL-MIL [ 27 ]	WACV’23	I3D RGB	85.63
MGFN † [ 5 ]	AAAI’23	I3D RGB	82.79
PE-MIL [ 4 ]	CVPR’24	I3D RGB	86.83

Table 4: Weakly supervised anomaly detection on UCF-Crime. Frame-level AUC (%). We reproduce all baselines for which official code is publicly available ( † ); remaining methods are listed with their originally reported numbers. Best in bold , second best underlined .

Configuration	AUC $\uparrow$	AP $\uparrow$	F1 $\uparrow$	tIoU $\uparrow$
w/o effect supervision	0.419	0.264	0.139	0.329
w/o bag objective	0.754	0.518	0.309	0.335
w/o soft targets	0.453	0.283	0.133	0.326
w/o multi-scale candidates	0.504	0.314	0.040	0.098
Full CoRE	0.744	0.514	0.364	0.429

Table 5: Component ablation on DoTA. Each row removes one component of CoRE ; the full configuration is shown in the last row. Best per column in bold .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	RISEE [ 40 ]	DoTA [ 42 ]	UCF-Crime [ 34 ]
Training supervision	Perceived-risk score	Video-level normal/abnormal	Video-level normal/abnormal
Evaluation protocol	5-fold scenario CV	Released 1,140-video test split	Standard 290-video test split
Input representation	ResNet-50; object track features	ResNet-50 / CLIP ViT-B/32	Ten-crop I3D RGB
Support candidates	16 positions; tracked objects	Temporal windows $\{1,2,4\}$	32 segments; windows $\{1,3,5\}$
Default intervention	Local mean	Local mean	Video-mean replacement
Target form	Competitive support	Competitive support	Independent temporal support

Appendix

Table 6: Experimental setup for CoRE. Dense temporal or object labels are not used during training.

Support	Train $\rightarrow$ Eval	Sel. Drop $\uparrow$	Gain vs. Rand. $\uparrow$
Temporal	Mean $\rightarrow$ Mean	0.0826	0.0558
Temporal	Blur $\rightarrow$ Mean	0.0810	0.0542
Temporal	Mean $\rightarrow$ Blur	0.0706	0.0418
Temporal	Blur $\rightarrow$ Blur	0.0705	0.0417
Object	Mean $\rightarrow$ Mean	0.0615	0.0284
Object	Blur $\rightarrow$ Mean	0.0588	0.0257

Appendix

Table 7: Cross-operator robustness of CoRE support learning on RISEE. We report the selection metrics shared by temporal and object support.

Support	Effect $\rho$ $\uparrow$	Agree@1 $\uparrow$	Top-3 Jaccard $\uparrow$
Temporal	0.7344	0.6089	0.6765
Object	0.7356	0.8333	0.8512

Appendix

Table 8: Agreement between local-mean and blur intervention effects.

Teacher	MAE $\downarrow$	Spearman $\uparrow$	Sel. Drop $\uparrow$	Gain vs. Rand. $\uparrow$	Effect $\rho$ $\uparrow$
Mean Pool	0.6139	0.6966	0.8519	0.5575	0.7956
Attention-MIL	0.5389	0.7241	2.4585	1.7356	0.6028
Canonical	0.5236	0.7343	2.5256	1.6813	0.7417

Appendix

Table 9: Robustness of CoRE to the choice of coarse risk teacher on RISEE. Only the teacher used to construct intervention effects is changed.

Support	Sel. Drop $\uparrow$	Gain vs. Rand. $\uparrow$	Effect $\rho$ $\uparrow$	NDCG@3 $\uparrow$	Top Perc. $\uparrow$
Attention-MIL attention	2.3295	1.6066	0.5683	0.8211	0.8823
CoRE distilled support	2.4585	1.7356	0.6028	0.8549	0.9082

Appendix

Table 10: Direct Attention-MIL attention versus CoRE support distilled from intervention effects of the same Attention-MIL teacher.

Component	Mean (ms) $\downarrow$	p95 (ms) $\downarrow$	Peak GPU (MB) $\downarrow$
Coarse teacher head	2.0679	2.0836	82.87
Temporal student head	2.0676	2.0833	82.87
Object student head	0.5510	0.5663	175.92
ResNet-50, 16 frames	10.8651	10.8832	260.82
Offline temporal target construction	14.2064	14.2444	88.10
Naive intervention explanation	14.2172	14.2654	88.10

Appendix

Table 11: Runtime on an NVIDIA H100 NVL. Intervention-based target construction is performed offline; CoRE uses direct student inference at test time.

Weakly Supervised Learning

Weakly supervised learning trains models from incomplete, noisy, coarse, or indirect labels instead of dense ground truth.

See all 156 papers

Towards Driver Behavior Understanding: Weakly-Supervised Risk Perception in Driving Scenes
Mar 6, 2026Nakul Agarwal, Yi-Ting Chen, Behzad DariushAutonomous Driving Datasets Object Detection
Honda Research Institute USA, Inc. · National Yang Ming Chiao Tung University.
RiskCueBench: Benchmarking Anticipatory Reasoning from Early Risk Cues in Video-Language Models
Jan 6, 2026Sha Luo, Yogesh Prabhu, Timothy Ossowski +2Temporal Video Understanding Temporal Reasoning in VLMs
University of Wisconsin–Madison, Madison, WI, USA · University of California San Diego, San Diego, CA, USA
Two-Pass Zero-Shot Temporal-Spatial Grounding of Rare Traffic Events in Surveillance Video
May 2, 2026Jiantang HuangTemporal Video Grounding Large Vision-Language Models

Mar 6, 2026J/K move · Enter open · S save

Towards Driver Behavior Understanding: Weakly-Supervised Risk Perception in Driving Scenes

Nakul Agarwal, Yi-Ting Chen, Behzad Dariush

Honda Research Institute USA, Inc. · National Yang Ming Chiao Tung University.

Autonomous Driving Datasets Object Detection

Open paper AskPDF arXiv

Achieving zero-collision mobility remains a key objective for intelligent vehicle systems, which requires understanding driver risk perception-a complex cognitive process shaped by voluntary response of the driver to external stimuli and the attentiveness of surrounding road users towards the ego-vehicle. To support progress in this area, we introduce RAID (Risk Assessment In Driving scenes)-a large-scale dataset specifically curated for research on driver risk perception and contextual risk assessment. RAID comprises 4,691 annotated video clips, covering diverse traffic scenarios with labels for driver's intended maneuver, road topology, risk situations (e.g., crossing pedestrians), driver responses, and pedestrian attentiveness. Leveraging RAID, we propose a weakly supervised risk object identification framework that models the relationship between driver's intended maneuver and responses to identify potential risk sources. Additionally, we analyze the role of pedestrian attention in estimating risk and demonstrate the value of the proposed dataset. Experimental evaluations demonstrate that our method achieves 20.6% and 23.1% performance gains over prior state-of-the-art approaches on the RAID and HDDS datasets, respectively.

Abstract

Weakly Supervised Learning

Explore similar work

Towards Driver Behavior Understanding: Weakly-Supervised Risk Perception in Driving Scenes

RiskCueBench: Benchmarking Anticipatory Reasoning from Early Risk Cues in Video-Language Models

Two-Pass Zero-Shot Temporal-Spatial Grounding of Rare Traffic Events in Surveillance Video