Towards Driver Behavior Understanding: Weakly-Supervised Risk Perception in Driving Scenes

Nakul Agarwal; Yi-Ting Chen; Behzad Dariush

cs.CVMar 6, 2026

Towards Driver Behavior Understanding: Weakly-Supervised Risk Perception in Driving Scenes

Authors: Nakul Agarwal, Yi-Ting Chen, Behzad Dariush

Organizations: Honda Research Institute USA, Inc. · National Yang Ming Chiao Tung University.

Abstract

Achieving zero-collision mobility remains a key objective for intelligent vehicle systems, which requires understanding driver risk perception-a complex cognitive process shaped by voluntary response of the driver to external stimuli and the attentiveness of surrounding road users towards the ego-vehicle. To support progress in this area, we introduce RAID (Risk Assessment In Driving scenes)-a large-scale dataset specifically curated for research on driver risk perception and contextual risk assessment. RAID comprises 4,691 annotated video clips, covering diverse traffic scenarios with labels for driver's intended maneuver, road topology, risk situations (e.g., crossing pedestrians), driver responses, and pedestrian attentiveness. Leveraging RAID, we propose a weakly supervised risk object identification framework that models the relationship between driver's intended maneuver and responses to identify potential risk sources. Additionally, we analyze the role of pedestrian attention in estimating risk and demonstrate the value of the proposed dataset. Experimental evaluations demonstrate that our method achieves 20.6% and 23.1% performance gains over prior state-of-the-art approaches on the RAID and HDDS datasets, respectively.

Figures & tables

Fig. 1 : Perception of risk is a complex cognitive process that is manifested, among other things, by a voluntary response of the driver to external stimuli (e.g. deviating from the planned path in response to a truck that is blocking the path) as well as the apparent attentiveness of participants in the scene (e.g. crossing cyclist that is not attentive to the ego-vehicle).

	Videos	Avg. Video Duration (s)	Risk Objects	Risk Situations	Driver Action	Driver Response	CAN Data	LiDAR	Road Topology	Pedestrian Attention	Face Annotations	Pedestrian Tracklets
JAAD [ 5 ]	346	5-15	✗	✗	✗	✗	✗	✗	✗	✓	✗	✓
PIE [ 6 ]	53	600	✗	✗	✗	✗	✓	✗	✓	✓	✗	✓
STIP [ 7 ]	556	-	✗	✗	✗	✗	✗	✗	✗	✗	✗	✗
LOKI [ 8 ]	644	13	✗	✗	✓	✗	✓	✓	✗	✗	✗	✓
OATS [ 9 ]	1026	20	✗	✗	✓	✗	✓	✓	✗	✗	✗	✓
DRAMA [ 10 ]	17785	2	17066	-	✓	✗	✓	✗	✗	✗	✗	✗

TABLE I: Comparison of our proposed RAID with other driving scene datasets. ’-’ indicates the trait is present but not quantified/categorized in the dataset; ’✗’ indicates the trait is absent.

Fig. 2 : Annotation statistics of our RAID dataset.

Fig. 3 : Proposed Network Architecture. The algorithm takes as input a sequence of RGB frames and object tracklets. We then extract corresponding agent-level features using partial convolution and RoIAlign, where in each iteration of the network, the partial convolution removes an agent using a binary mask. These features then form nodes of the graph convolution network. In parallel, the RGB frames are also used to obtain driver’s action using a temporal encoder-decoder LSTM network. Finally, the feature representation from the graph and driver’s action are combined to predict the driver’s response.

Method	Crossing Vehicle	Crossing Pedestrian	Parked Vehicle	Cong- estion	Avg mAcc
Random Selection	14.78	6.32	7.21	6.74	8.76
Driver’s Attention [ 18 ]	16.80	8.90	10.00	21.30	14.25
Driving Model [ 15 ]	25.40	19.88	21.02	18.58	21.22
Object-level Attention [ 24 ]	26.52	17.50	22.20	45.05	27.81
DROID [ 12 ]	27.50	13.60	26.00	51.30	29.60
DROID [ 12 ] ‡	32.50	12.90	28.40	57.50	32.83

TABLE II : Risk object identification results on the HDDS dataset. The best performances are shown in bold. Method with ‡ includes driver action.

Method	Crossing Pedestrian	Crossing Vehicle	Car Blocking Ego Lane	Cong- estion	Cut- In	Jay- walking	Traffic Light	Stop Sign	Avg mAcc
Random Selection	5.90	9.56	7.38	7.42	10.26	4.40	2.94	5.67	6.99
Driving Model [ 15 ]	13.96	24.01	15.68	30.46	33.66	11.28	4.32	8.00	17.67
Object-level Attention [ 24 ]	13.27	24.17	14.42	41.58	32.30	12.37	2.02	6.53	18.33
Ours	15.64	31.05	34.48	27.35	27.69	3.42	2.83	11.76	19.28
Ours+	13.89	36.13	11.67	54.58	38.58	16.06	0.92	4.93	22.10

TABLE III : Risk object identification results on the RAID dataset. The best performances are shown in bold.

Method	Driver Action				Driver Response			Risk
Method	Left	Right	Straight	mAP	Continue	Alter	mAP	mAcc
Ours	-	-	-	-	68.75	93.22	80.98	19.28
Ours+	35.32	61.48	78.96	58.59	77.34	96.43	86.88	22.10

TABLE IV : The average precision (AP%) results on the different tasks of driver action and driver response.

	Method	Looking	Not Looking	mAP
Cls	Body [ 5 ]	46.08	78.14	62.10
Cls	Face	75.44	92.09	83.76
Det	Face	69.83	36.03	52.93

TABLE V : The average precision (AP%) of the classification (Cls) and detection (Det) results for pedestrians’ attention.

Panel of Figure 9 — Fig. 4 : Risk object identification results on the RAID dataset. The ground truth is shown in red boxes, predictions are shown using green boxes. The ego-vehicle is depicted by the orange car, and blue arrow shows future motion direction. A birds-eye-view representation is presented below each front-view image providing information including scene layout, intentions of traffic participants, and the ego vehicle.

Fig. 5 : Joint risk assessment on RAID . Top row shows detected agents with colored boxes and bottom bar chart uses matching box colors for risk scores. The black line marks the predicted ‘Continue’ score without intervention, and $\star$ shows adjusted scores after factoring in pedestrian attention.

Autonomous Driving Datasets

Autonomous driving datasets collect and organize driving-scene data to train and evaluate perception, prediction, planning, and safety systems.

See all 141 papers

CoRE: Weakly Supervised Coarse-to-Fine Risk Evidence Learning in Driving Videos
Aug 26, 2026Kaiser Hamid, Can Cui, Nade LiangWeakly Supervised Learning Weakly Supervised Video Anomaly Detection
Texas Tech University · Purdue University
Fusing Driver Perceived and Physical Risk for Safety Critical Scenario Screening in Autonomous Driving
Mar 7, 2026Chen Xiong, Ziwen Wang, Deqi Wang +4Autonomous Driving Safety Evaluation
Guangdong Provincial Key Laboratory of Intelligent Transportation System, School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China · school of Transportation and Logistics, Southwest Jiaotong University, Chengdu, China
DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios
May 16, 2026Sainithin Artham, Shankar Gangisetty, Avijit Dasgupta +1VLMs for Autonomous Driving Autonomous Driving Safety Evaluation
IIIT-Hyderabad, India

Aug 26, 2026J/K move · Enter open · S save

CoRE: Weakly Supervised Coarse-to-Fine Risk Evidence Learning in Driving Videos

Kaiser Hamid, Can Cui, Nade Liang

Texas Tech University · Purdue University

Weakly Supervised Learning Weakly Supervised Video Anomaly Detection

Open paper AskPDF arXiv

Perceived risk in driving evolves over time and may be supported by specific scene entities, yet supervision is typically limited to coarse video-level judgments. Learning when supporting evidence emerges and which entities support a risk predictor would ordinarily require costly temporal- and entity-level annotations. We introduce CoRE, a weakly supervised coarse-to-fine framework that learns fine-grained prediction support from coarse video supervision. CoRE first trains a video-level predictor and then freezes it. Structured interventions over candidate temporal regions or entity tracks measure how each candidate changes the coarse prediction, producing graded prediction-effect targets. These targets are distilled into a student that directly predicts temporal and entity support from the original video, without requiring interventions at inference. We evaluate this learning principle across three complementary settings: RISEE tests perceived-risk support from subjective clip-level judgments without temporal or entity-level risk annotations; DoTA provides independent temporal event annotations for evaluating weakly supervised traffic-anomaly localization; and UCF-Crime tests whether the same coarse-to-fine mechanism extends to a standard non-driving anomaly-detection benchmark. Across these settings, CoRE learns informative fine-grained support from coarse supervision, with strong temporal localization on DoTA and competitive performance on UCF-Crime. These results show that coarse video predictions can provide useful supervision for recovering the fine-grained evidence supporting them, without requiring corresponding fine-grained labels.

Abstract

Autonomous Driving Datasets

Explore similar work

CoRE: Weakly Supervised Coarse-to-Fine Risk Evidence Learning in Driving Videos

Fusing Driver Perceived and Physical Risk for Safety Critical Scenario Screening in Autonomous Driving

DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios