Predicting Risks Before They Happen: Vision-Language Reasoning for Autonomous Driving
Organizations: School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China
Abstract
% Waiting until a driving hazard becomes directly visible can leave an autonomous vehicle little time to respond. Observable precursors, such as another vehicle's unexpected braking, can provide evidence of a hidden hazard and an opportunity for earlier risk mitigation. However, existing driving datasets and language reasoning formulations provide limited supervision for connecting these precursors to potential risks and appropriate responses. We introduce PotentialRiskQA, an ego-view driving dataset that marks precursor and risk appearance times and provides structured annotations linking scene evidence, potential risks, and mitigation actions. Building on this dataset, PR-Reasoner trains vision-language models to generate a precursor-centered reasoning chain through scene-level supervision and auxiliary tasks for scene understanding, precursor identification, risk inference, driving suggestions, and meta-actions. Experiments across multiple backbones show improved scene understanding, risk cognition, and action quality, with fewer unsupported risk claims on most models. Ablations show that supervising intermediate reasoning steps further reduces unsupported risk claims and improves action quality beyond scene-level training alone. On NAVSIM, the trained representations improve open-loop planning performance, demonstrating benefits beyond language-based evaluation.
Figures & tables
| Group | VLMs | Format | SceUnd | RiskCog | FalseRisk | Action |
| Baseline | In-1B | 0.774 0.034 | 0.244 0.037 | 0.256 0.073 | 0.051 0.038 | 0.327 0.038 |
| In-2B | 0.641 0.047 | 0.187 0.035 | 0.265 0.081 | 0.094 0.056 | 0.274 0.030 | |
| In-4B | 0.662 0.051 | 0.250 0.040 | 0.291 0.085 | 0.098 0.060 | 0.504 0.042 | |
| In-8B | 0.829 0.043 | 0.297 0.041 | 0.321 0.090 | 0.128 0.073 | 0.359 0.044 | |
| Qw-2B | 0.842 0.047 | 0.226 0.044 | 0.251 0.082 | 0.017 0.017 | 0.475 0.045 | |
| Qw-4B | 0.769 0.060 | 0.304 0.047 | 0.222 0.085 | 0.094 0.064 | 0.414 0.038 |
| Backbone | Setting | RiskCog | FalseRisk | Action |
| In-4B | Direct risk CoT | 0.269 | 0.098 | 0.444 |
| Think-Driver CoT | 0.329 | 0.175 | 0.290 | |
| PR prompt | 0.291 | 0.098 | 0.504 | |
| PR-Reasoner | 0.368 | 0.162 | 0.832 | |
| Qw-8B | Direct risk CoT | 0.338 | 0.184 | 0.577 |
| Think-Driver CoT | 0.346 | 0.139 | 0.549 |
| Model | Supervision | Format | SceUnd | RiskCog | FalseRisk | Action | Scene understanding | Driving action | ||||
| Key objects | Risk info. | No halluc. | Safety | Efficiency | Executability | |||||||
| In-4B | Baseline | 0.662 | 0.250 | 0.291 | 0.098 | 0.504 | 0.470 | 0.128 | 0.274 | 0.423 | 0.701 | 0.487 |
| In-4B | Scene only | 0.927 | 0.350 | 0.299 | 0.068 | 0.803 | 0.560 | 0.286 | 0.269 | 0.744 | 0.808 | 0.859 |
| In-4B | Scene + auxiliary | 0.927 | 0.401 | 0.368 | 0.162 | 0.832 | 0.603 | 0.321 | 0.359 | 0.752 | 0.876 | 0.889 |
| Qw-8B | Baseline | 0.761 | 0.349 | 0.359 | 0.043 | 0.595 | 0.568 | 0.274 | 0.282 | 0.547 | 0.658 | 0.611 |
| Qw-8B | Scene only | 0.983 | 0.417 | 0.376 | 0.098 | 0.721 | 0.632 | 0.363 | 0.308 | 0.645 | 0.761 | 0.778 |
| Model | Pretrain | Format | SceUnd | RiskCog | FalseRisk | Action |
| In-4B | No | 0.919 | 0.341 | 0.321 | 0.077 | 0.762 |
| Yes | 0.927 | 0.401 | 0.368 | 0.162 | 0.832 | |
| Qw-8B | No | 0.962 | 0.399 | 0.416 | 0.147 | 0.795 |
| Yes | 0.957 | 0.483 | 0.389 | 0.261 | 0.807 |
| Window | Model | Format | SceUnd | RiskCog | FalseRisk | Action | Time (s) |
| 1 s | In-4B | 0.932 | 0.359 | 0.393 | 0.064 | 0.823 | 0.99 |
| Qw-8B | 0.863 | 0.402 | 0.444 | 0.107 | 0.809 | 1.64 | |
| 2 s | In-4B | 0.927 | 0.401 | 0.368 | 0.162 | 0.832 | 1.49 |
| Qw-8B | 0.957 | 0.483 | 0.389 | 0.261 | 0.807 | 2.35 | |
| 3 s | In-4B | 0.893 | 0.363 | 0.338 | 0.081 | 0.803 | 1.97 |
| Qw-8B | 0.966 | 0.447 | 0.363 | 0.103 | 0.815 | 3.15 |
| Category | RiskCog | FalseRisk | Action | ||||
| Raw | PR | Raw | PR | Raw | PR | ||
| BZ | 153 | 0.272 | 0.453 | 0.068 | 0.200 | 0.576 | 0.801 |
| SI | 75 | 0.444 | 0.396 | 0.116 | 0.111 | 0.592 | 0.799 |
| SB | 72 | 0.333 | 0.380 | 0.000 | 0.037 | 0.569 | 0.795 |
| BZ_C | 55 | 0.406 | 0.291 | 0.061 | 0.079 | 0.559 | 0.754 |
| OD | 40 | 0.608 | 0.667 | 0.100 | 0.233 | 0.578 | 0.790 |
| VLM setting | NC | DAC | TTC | Comfort | EP | PDMS | ADE | FDE |
| Direct risk CoT | 92.26 | 78.92 | 83.62 | 100.00 | 64.75 | 67.11 | 0.844 | 2.201 |
| PR prompt | 92.90 | 78.22 | 85.35 | 100.00 | 63.61 | 67.13 | 0.842 | 2.190 |
| PR-Reasoner | 94.41 | 81.52 | 87.95 | 100.00 | 66.81 | 71.09 | 0.801 | 2.076 |
| Output setting | In-4B | Qw-8B |
| Direct meta-action | 1.15 (1.00 ) | 2.02 (1.00 ) |
| Direct risk CoT | 1.28 (1.11 ) | 2.15 (1.06 ) |
| Think-Driver-style CoT | 1.33 (1.16 ) | 2.11 (1.04 ) |
| Full PR-Reasoner chain | 1.49 (1.30 ) | 2.35 (1.16 ) |
VLMs for Autonomous Driving
Vision-language models for autonomous driving connect camera and sensor data with language to interpret scenes, predict hazards, and plan actions.
See all 220 papers