Organizations: Zhongguancun Laboratory, Beijing, China · Tianjin University, Beijing, China · Hong Kong Polytechnic University, Hong Kong, China · Nanyang Technological University, Singapore · Beihang University, Beijing, China
Autonomous driving risk identification aims to determine which observed object is likely to become safety-critical to the ego vehicle. Existing approaches typically predict scene-level accidents, infer risk objects indirectly from ego behavior, or apply geometric checks after trajectory forecasting, without directly using predicted ego--object relations for risk-source localization. We propose RiskWorld, an object-centric latent world model that identifies risk from the imagined evolution of each candidate relative to the ego vehicle. RiskWorld combines pretrained predictive video representations with structured ego--object histories, contextualizes observed interactions, and rolls relation-aware object states into the future using RSSM-style latent dynamics. It decodes the rollout into object-level risk scores, supported by auxiliary future-relation and temporal-risk predictions. Inference uses only observations up to the current time, while logged futures provide training supervision. On RiskBench, RiskWorld achieves the best overall F1 of 63.0% and the lowest false-alarm rate of 2.1%. Further analyses show that the learned rollout captures the evolution of object-level risk before critical events, while RiskWorld's selections preserve planning-critical information under filtered observation.
Figures & tables
Figure 1: Existing approaches predict scene-level risk, infer risk objects from ego behavior, or apply post-hoc geometric checks. RiskWorld instead predicts object-conditioned future ego–object relations and uses them to identify the risk source.
Figure 2: Overview of RiskWorld. Observed RGB frames, object tracks, and ego motion are encoded into relation-aware object states, rolled forward with object-wise RSSM-style latent dynamics, and decoded into risk-source scores with auxiliary future-relation and temporal-risk predictions. Future annotations supervise training only; inference uses observed history.
Interactive
Collision
Obstacle
Non-inter.
All
Method
P(%) ↑
R(%) ↑
PIC ↓
P(%) ↑
R(%) ↑
PIC ↓
P(%) ↑
R(%) ↑
PIC ↓
FA(%) ↓
F1 ↑
Random
9.5
50.0
16.7
19.2
49.8
15.7
7.2
50.3
11.4
80.8
15.1
Range (5m)
38.7
10.7
20.3
85.6
27.7
4.1
45.0
18.3
10.2
3.3
30.0
Range (10m)
43.6
69.1
7.1
69.3
62.4
3.8
33.3
88.5
0.8
15.2
53.6
Kalman filter
37.0
54.0
15.4
59.9
58.1
4.2
35.3
61.2
10.2
18.8
46.7
Social-GAN
39.4
57.7
14.1
60.8
45.4
7.8
36.9
69.3
4.0
16.3
46.0
Table 1: Risk localization and anticipation results. P, R, PIC, and FA denote precision, recall, Progressive Increasing Cost, and false-alarm rate. F1 is micro-averaged over all scenario types. ∗ denotes our object-level adaptations of BADAS and FLaRA.
Figure 3: Risk-localization behavior and operating point. (a–b) Test-set recall and false-positive rate across ego–object distances for RiskWorld and fixed-range rules. (c) Overall, interactive, and collision F1 together with non-interactive false-alarm rate across validation thresholds; the marker denotes the selected η=0.70 used for all test categories.
Figure 4: Future-risk prediction and representation. (a) Horizon-wise Brier score on risk-source objects, comparing the learned rollout with current-state persistence. Lower is better. (b) t-SNE of 1,500 balanced decision representations from risk sources and hard non-risk objects.
Figure 5: Temporal risk separation. Mean scores of the annotated risk source and the strongest non-risk object from 3 s before the critical frame; shaded regions denote 95% confidence intervals.
Variant
I ↑
C ↑
O ↑
N ↓
All ↑
w/o world features
47.6
60.0
56.7
21.3
50.6
w/o object visual token
55.7
56.6
51.9
3.3
54.6
w/o relation attention
63.8
45.9
60.7
1.2
54.3
Deterministic GRU
65.3
63.9
68.5
6.0
62.0
w/o future rollout
62.4
59.9
64.4
5.9
60.2
w/o future supervision
68.5
57.5
65.4
4.4
61.6
Table 2: Ablation of the main RiskWorld components. I, C, and O report F1 on interactive, collision, and obstacle scenarios; N reports the false-alarm rate on non-interactive scenes; All is overall micro-F1. All values are percentages.
Interactive
Obstacle
Method
IR ↓
CR(%) ↓
IR ↓
CR(%) ↓
Auto-pilot
0.45
33.6
0.52
58.7
Full observation
0.00
0.0
0.00
0.0
Ground-truth risk
0.02
0.4
0.04
5.3
Random
0.11
7.7
0.44
48.1
Range (10m)
0.01
6.2
0.24
26.4
Table 3: Planning-aware evaluation under the filtered-observation protocol. Lower values indicate that the selected object better preserves planning-critical information.
Figure 6: Risk identification and temporal evolution. The upper panels mark the predicted risk source in red. The lower panels show the annotated source score (orange) and maximum candidate score (cyan) from −3 s to +1 s relative to the critical frame. Black and red vertical lines indicate the displayed and critical frames, respectively.
Safe motion planning in automated driving requires anticipating evolving traffic risks and deciding when to revise the current planned trajectory. We introduce RiskWorld, a risk-aware world modeling framework for shared occupancy forecasting and selective trajectory replacement. Spatial risk fields and temporal actor context are fused with visual bird's-eye-view features. Flow-guided evolution transports occupancy and scene features, while signed residuals correct occupancy after transport. One forecast is generated per planning step and reused across candidates. Each candidate is compared with a current-state persistence reference, yielding a nonnegative collision-score correction. The trajectory selected by current-world evaluation serves as the planning anchor and is replaced only when additional predicted risk triggers intervention and an alternative satisfies component-wise constraints on predicted risk and trajectory error. Candidate geometries remain unchanged. We evaluate RiskWorld for open-loop planning on nuScenes using camera features, annotation-derived current and historical actor states, and dataset-provided map context. RiskWorld achieves the lowest collision rate at a long evaluation horizon of 3 s, and the second-best average L2 error among various state-of-the-art baselines, while running at 11.5 FPS on a single NVIDIA RTX 4090 with 90.81 M parameters. Within-setting ablations show that RiskWorld achieves lower collision rates than the current-state rescoring baseline, while forecast reuse enables additional candidates to be evaluated at low marginal computational cost.
Rongxiang Zeng, Linsen Cai, Jiafu Zhang +7
RWTH Aachen University, Aachen, Germany · Monash University, Melbourne, Australia · Technical University of Munich (TUM), Munich, Germany
Comprehensive situational awareness is essential for autonomous vehicles operating in safety-critical environments, as it enables the identification and mitigation of potential risks. Although recent Multimodal Large Language Models (MLLMs) have shown promise on general vision-language tasks, our findings indicate that zero-shot MLLMs still underperform compared to domain-specific methods in fine-grained, spatially grounded risk assessment. To address this gap, we propose DriveSafe, a framework for risk-aware scene understanding that leverages structured natural language descriptions. Specifically, our method first generates spatially grounded captions enriched with multimodal context, including motion, spatial, and depth cues. These captions are then used for downstream risk assessment, explicitly identifying hazardous objects, their locations, and the unsafe behaviors they imply, followed by actionable safety suggestions. To further improve performance, we employ caption-risk pairings to fine-tune a lightweight adapter module, efficiently injecting domain-specific knowledge into the base LLM. By conditioning risk assessment on explicit language-based scene representations, DriveSafe achieves significant gains over both zero-shot MLLMs and prior domain-specific baselines. Exhaustive experiments on the DRAMA benchmark demonstrate state-of-the-art performance, while ablation studies validate the effectiveness of our key design choices. Project page: https://cvit.iiit.ac.in/ research/projects/cvit-projects/drivesafe
Safety is a fundamental requirement for autonomous driving, yet existing end-to-end driving models still lack explicit risk-aware learning capacities. Existing rule-based risk models provide interpretable safety priors, yet their absolute risk scores depend on handcrafted functions, coefficients, and thresholds. Learning-based risk representations reduce part of this manual design, but their supervision often relies on occupancy-derived labels or heuristic cost values, which may not capture ego-conditioned planning risk. In this paper, we propose DRiF, a data-driven risk-field framework for safer end-to-end autonomous driving. DRiF learns a shared BEV feature with static map segmentation, dynamic risk prediction, and vehicle planning. For dynamic risk learning, DRiF converts rule-based safety priors into pairwise risk labels, and trains the risk field to preserve relative risk ordering instead of regressing handcrafted absolute scores. Experiments on Bench2Drive show that DRiF achieves competitive overall performance, with consistent improvements in driving score, success rate, and collision-related metrics. These results establish relative risk supervision as an effective way to connect explicit safety structure with end-to-end planning. The data and code will be publicly available.