STRIDE: Evaluating Spatiotemporal Reasoning in Driving Edge Cases
Organizations: University of California, Los Angeles · Toyota InfoTech Labs
Abstract
In autonomous driving, perceiving surrounding objects is not enough: failing to understand how they move and interact over time can lead to dangerous decisions. Evaluating vision-language models (VLMs) therefore requires testing whether they connect past spatial observations to future consequences. Yet existing benchmarks often assess edge cases through basic perception questions or test temporal understanding in simple scenarios, leaving complex spatiotemporal reasoning underexplored. We introduce STRIDE (SpatioTemporal Reasoning In Driving Edge Cases), an object-centric benchmark for long-tail, safety-critical driving scenarios. STRIDE covers six task families: spatial perception, spatial understanding, temporal memory, temporal extrapolation, trajectory prediction, and scene-context awareness. It pairs multiple-choice questions probing individual spatiotemporal judgments with related free-form visual question answering (VQA) requiring detailed scene explanations and justifications of driving decisions. We prioritize scenes with unusual object behaviors and complex traffic interactions, generate reference answers from structured dataset annotations through automated procedures, and use limited language-model assistance and human verification to ensure annotation quality. Across 12 proprietary, open-weight, and driving-specialized models, even the best performer, GPT-6-Astra, achieves only 43.4% average multiple-choice accuracy, versus an approximately 20% random-choice baseline. Further analysis reveals that strong performance in describing driving scenes does not necessarily translate to accurate safety-critical judgments, and that models struggle more with reconstructing object motion than recalling past positions. STRIDE thus exposes spatiotemporal driving limitations that aggregate VQA scores can conceal.
Figures & tables
| Benchmark | Input | QA Format | Capabilities | |||||
|---|---|---|---|---|---|---|---|---|
| Format | View | Spatial | Temporal | Edge Cases | Grounding | Ego Traj. | ||
| NuScenes-QA ( Qian et al., 2024 ) | Image / Video | Surround | ||||||
| DriveLM-nuScenes ( Sima et al., 2024 ) | Image | Surround | ||||||
| Reason2Drive ( Nie et al., 2024 ) | Video | Mixed | ||||||
| LingoQA ( Marcu et al., 2024 ) | Video | Front | ||||||
| OmniDrive ( Wang et al., 2025 ) | Image | Surround | ||||||
| nuScenes | Waymo E2E | |||||||
| Model | MCQ | OEQ | ADE | FDE | MCQ | OEQ | ADE | FDE |
| Random Chance | 19.8 | – | – | – | 20.3 | – | – | – |
| Human Performance | 79 | – | – | – | 69 | – | – | – |
| Driving Expert Models | ||||||||
| Dolphins ( Ma et al., 2024 ) | 18.5 | 3.24 | 8.16 | 3.81 | 22.2 | 1.86 | 18.43 | 8.60 |
| UniAD 2.0 ( Hu et al., 2023 ) | – | – | 0.56 | 0.79 | – | – | – | – |
| (a) nuScenes | Model | Representative tasks | |||
|---|---|---|---|---|---|
| Capability | Task 1 | Score | Task 2 | Score | |
| Spatial perception | G-Pro | SP-3: Distance Estimate | 64% | SP-7: Spatial desc. | 2.10 |
| Spatial understanding | GPT-5.5 | SC-6: Scene desc. | 9.22 | SU-2: Drivable region | 22% |
| Temporal memory | GPT-5.5 | TM-2: Prev. position | 72% | TM-3: Recent motion | 18% |
| Temporal extrapolation | Best | TE-1: Future motion | 40% | TE-6: Scene evolution | 6.98 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Pearson | Spearman | Mean |
|---|---|---|---|
| BLEU | |||
| METEOR | |||
| BLEURT | — | ||
| Qwen3-VL-32B (text-only) | |||
| GPT-5.5 (vision) | |||
| Qwen3-VL-32B (vision) |
| Template | Question | Reason for removal |
|---|---|---|
| SC-1 | Weather and environmental condition | No suitable choice for dark scenes. |
| SC-2 | Time of day | Samples are mostly daytime, so the label is not reliable. |
| SC-3 | Road type and driving environment | Parked cars do not imply a parking lot, and the sample contains no highway. |
| TE-3 | Future lane of the target object | The frames do not provide enough lane information. |
| TE-4 | Ego response to the target object | Ground truth is incorrect. |
| TE-5 | Next traffic event | Ground truth is incorrect and the question requires lane identification. |
| Task family | Questions | MCQ | OEQ | Scenes | Question IDs |
|---|---|---|---|---|---|
| Scene context | 50 | 0 | 50 | 50 | SC-6 |
| Spatial perception | 250 | 200 | 50 | 242 | SP-1, SP-2, SP-3, SP-5, SP-7 |
| Spatial understanding | 200 | 150 | 50 | 186 | SU-1, SU-2, SU-4, SU-7 |
| Temporal memory | 200 | 200 | 0 | 187 | TM-2, TM-3, TM-4, TM-5 |
| Temporal extrapolation | 100 | 50 | 50 | 100 | TE-1, TE-6 |
| Trajectory | 350 | 200 | 150 | 316 | TRJ-1–TRJ-7 |
| Task family | Questions | MCQ | OEQ | Scenes | Question IDs |
|---|---|---|---|---|---|
| Scene context | 200 | 150 | 50 | 117 | SC-1, SC-2, SC-3, SC-4 |
| Spatial perception | 150 | 100 | 50 | 108 | SP-2, SP-3, SP-3a |
| Temporal memory | 150 | 150 | 0 | 109 | TM-2, TM-3, TM-5 |
| Temporal extrapolation | 200 | 200 | 0 | 158 | TE-1, TE-2, TE-4, TE-5 |
| Trajectory | 500 | 300 | 200 | 135 | TRJ-1–TRJ-10 |
| Split total | 1200 | 900 | 300 | 335 | 24 IDs |
| Model | Identifier | Reasoning / thinking |
| Proprietary LVLMs | ||
| GPT-5.5 | gpt-5.5 | reasoning_effort=low |
| GPT-5.6 | gpt-5.6 | reasoning_effort=low |
| GPT-6-Astra | gpt-6-Astra | reasoning_effort=low |
| Gemini-3.6-Flash | gemini-3.6-flash | thinking_level=low |
| Gemini-3.1-Pro | gemini-3.1-pro-preview | thinking_level=low |
| Proprietary | Open-weight | Driving Expert | ||||||||||||
| ID | Task | Metric | GPT-5.5 | GPT-5.6 | GPT-6 | G-Fl. | G-Pro | Claude | Q35 | Q2.5 | Q30-T | Q8-T | Q27 | Dolphins |
| Scene Context | ||||||||||||||
| SC-6 | Environment description | Judge | 9.22 | 8.94 | 8.94 | 8.50 | 7.22 | 8.94 | 8.84 | 7.16 | 9.14 | 9.10 | 8.94 | 4.58 |
| Space Perception | ||||||||||||||
| SP-1 | Trajectory direction | Acc. (%) | 24.0 | 32.0 | 38.0 | 36.0 | 24.0 | 16.0 | 14.0 | 20.0 | 20.0 | 18.0 | 6.0 | 26.0 |
| SP-2 | Relative position | Acc. (%) | 56.0 | 42.0 | 60.0 | 62.0 | 54.0 | 40.0 | 46.0 | 32.0 | 40.0 | 34.0 | 52.0 | 10.0 |
| Proprietary | Open-weight | Driving Expert | |||||||||||||
| ID | Task | Metric | GPT-5.5 | GPT-5.6 | GPT-6 | G-Fl. | G-Pro | Claude | Q35 | Q2.5 | Q30-T | Q8-T | Q8-I | Q27 | Dolphins |
| Scenario / Corner Case | |||||||||||||||
| SC-1 | Scene description | Acc. (%) | 44.0 | 46.0 | 48.0 | 40.0 | 38.0 | 40.0 | 32.0 | 42.0 | 36.0 | 38.0 | 44.0 | 40.0 | 32.0 |
| SC-2 | Main risk source | Acc. (%) | 28.0 | 24.0 | 36.0 | 20.0 | 30.0 | 36.0 | 22.0 | 24.0 | 26.0 | 34.0 | 26.0 | 30.0 | 20.0 |
| SC-3 | Preferred ego action | Acc. (%) | 48.0 | 46.0 | 44.0 | 42.0 | 44.0 | 38.0 | 36.0 | 46.0 | 38.0 | 42.0 | 38.0 | 44.0 | 26.0 |
| SC-4 | Scene/risk/action explanation | Judge | 8.08 | 7.80 | 7.76 | 7.46 | 4.22 | 8.28 | 7.14 | 4.02 | 6.92 | 6.98 | 7.40 | 7.76 | 2.38 |
| ID | Fmt | Template | N | W |
|---|---|---|---|---|
| SC-1 | MCQ | What is the primary weather and environmental condition in the current scene? A Clear/Sunny; B Rain/Wet; C Overcast; D Snow/Ice; E Fog/Haze | ||
| SC-2 | MCQ | Based on the lighting conditions and sky visibility, what is the estimated time of day? A Daytime; B Nighttime; C Dawn/Dusk | ||
| SC-3 | MCQ | Which of the following best describes the overall road typology and driving environment? A Urban intersection; B Straight urban/suburban road; C Highway; D Construction zone; E Parking lot/off-road | ||
| SC-4 | MCQ | How spatially dense or crowded is the current traffic environment considering all visible road users? A Sparse; B Moderate; C Dense | ||
| SC-5 | MCQ | What is the baseline safety risk level of the overall scene, independent of any single specific object? A Low; B Moderate; C High | ||
| SC-6 | OEQ | Describe the overall environment of the scene, including weather, lighting conditions, road type, and notable surrounding activity. | Y |
| ID | Fmt | Template | N | W |
|---|---|---|---|---|
| SP-1 | MCQ | What is the trajectory direction of <obj> in relation to the ego vehicle? A Toward ego; B Away from ego; C Left-to-right; D Right-to-left; E Static; F Same direction as ego | Y | |
| SP-2 | MCQ | What is the spatial relationship between <obj> and ego vehicle? A In front; B Front-left; C Front-right; D Directly left; E Directly right; F Rear-left; G Rear-right; H Behind | Y | Y |
| SP-3 | MCQ | What is the approximate distance between <obj> and ego vehicle? A 0–5m; B 5–15m; C 15–30m; D 30–50m; E 50m | Y | Y |
| SP-4 | MCQ | Use the ego vehicle’s lanes as reference, which lane is <obj> located in? A Same lane; B Left adjacent; C Right adjacent; D Two lanes left; E Two lanes right | ||
| SP-5 | MCQ | What is the occlusion state of <obj> ? A Fully visible; B Partially occluded ( 50%); C Mostly occluded ( 50%); D Fully occluded but inferred | Y | |
| SP-6 | MCQ | Is <obj> on the drivable ground plane? A On road; B On sidewalk; C Above the ground |
| ID | Fmt | Template | N | W |
|---|---|---|---|---|
| SU-1 | MCQ | What physical constraints affect the ego vehicle given <obj> ? A Blocking ego lane; B Partially obstructing; C No constraint; D Dynamic obstruction in ego lane | Y | |
| SU-2 | MCQ | Which region around the ego vehicle is safely drivable? A Front only; B Front-left; C Front-right; D Left only; E Right only; F None; G All directions | Y | |
| SU-3 | MCQ | What is the risk level posed by <obj> to the ego vehicle? A Low; B Moderate; C High; D Critical/imminent collision | ||
| SU-4 | MCQ | Does the ego vehicle have sufficient clearance to change lanes? A Safe left; B Safe right; C Unsafe both; D Safe both | Y | |
| SU-5 | MCQ | Which maneuver is geometrically feasible at this moment? A Maintain lane; B Lane change left; C Lane change right; D Turn left; E Turn right; F Stop only | ||
| SU-6 | MCQ | How spatially dense is the environment around ego considering the number of objects and their distance to the ego? A Sparse; B Moderate; C Dense; D Highly congested |
| ID | Fmt | Template | N | W |
|---|---|---|---|---|
| TE-1 | MCQ | What is the most likely future motion of <obj> over the next few seconds? A Continue straight; B Turn left; C Turn right; D Stop; E Cross ego path | Y | Y |
| TE-2 | MCQ | If both vehicles maintain their current motion, when would a collision occur? A No collision; B 10s; C 5–10s; D 2–5s; E 2s | Y | |
| TE-3 | MCQ | Which lane will <obj> most likely occupy in the next few seconds? A Remain; B Left lane; C Right lane; D Enter ego lane; E Exit roadway | ||
| TE-4 | MCQ | What should be the ego vehicle’s primary response to <obj> ? A Maintain speed; B Slow down; C Prepare to stop; D Change lane; E Yield; F Emergency braking | Y | |
| TE-5 | MCQ | What traffic event is most likely to occur next? A Lane change; B Vehicle stopping; C Pedestrian crossing; D Vehicle merging; E No significant event | Y | |
| TE-6 | OEQ | Describe how the current scene is likely to evolve over the next few seconds and what actions the ego vehicle should take to maintain safety. | Y |
| ID | Fmt | Template | N | W |
|---|---|---|---|---|
| TM-1 | MCQ | Which objects previously appeared in the scene but are no longer visible in current frame? A None; B Pedestrian; C Vehicle; D Bicycle; E Static obstacle; F Multiple objects | ||
| TM-2 | MCQ | Where was <obj> located relative to the ego vehicle in the previous frame(s)? A In front; B Front-left; C Front-right; D Left; E Right; F Behind | Y | Y |
| TM-3 | MCQ | What was the recent motion pattern of <obj> ? A Moving straight; B Slowing down; C Accelerating; D Turning left; E Turning right; F Stopped previously | Y | Y |
| TM-4 | MCQ | How many objects are currently occluded but were previously visible in the previous frame? A None; B 1; C 2–3; D More than 3 | Y | |
| TM-5 | MCQ | In which lane was <obj> located earlier in the sequence? A Same lane; B Left adjacent; C Right adjacent; D Outside roadway | Y | Y |
| TM-6 | OEQ | Describe the recent history of the scene, including objects that appeared or disappeared, changes in their motion, and any interactions with the ego vehicle. |
| ID | Fmt | Template | N | W |
|---|---|---|---|---|
| TRJ-1 | MCQ | Based on the past 5 consecutive 360 ∘ multi-camera frames, what best describes the ego vehicle’s recent maneuver over those past frames? A Go straight; B Slightly veer left; C Slightly veer right; D Turn left; E Turn right; F Slow to near stop | Y | Y |
| TRJ-2 | MCQ | Based on the past 5 consecutive 360 ∘ multi-camera frames, what is the most likely ego-vehicle maneuver over the next 5 future frames (approximately 2.5 seconds)? A – F as TRJ-1 | Y | Y |
| TRJ-3 | MCQ | Based on the past 5 consecutive 360 ∘ multi-camera frames, at the 5th future frame (approximately 2.5 seconds ahead), which relative endpoint region best matches the ego vehicle trajectory in the current ego-centric coordinate system? A Front-left; B Front-center; C Front-right; D Left-side; E Right-side; F Near current position | Y | Y |
| TRJ-4 | MCQ | Based on the past 5 consecutive 360 ∘ multi-camera frames, how does the ego vehicle speed trend evolve over the next 5 future frames (approximately 2.5 seconds)? A Accelerating; B Constant speed; C Decelerating; D Stopping/nearly stopped | Y | Y |
| TRJ-5 | OEQ | Using past ego positions <trajectory_history> in the ego frame (current ), predict the next 5 future positions as a JSON array of 5 points. | Y | † |
| TRJ-6 | OEQ | Same setup as TRJ-5, with known first future point <first_future_position> . Predict only the remaining 4 points. | Y | † |
| ID | Fmt | Waymo template |
|---|---|---|
| SC-1 | MCQ | Which option best describes the current scene? A highway; B traffic intersection; C construction zone; D right turn |
| SC-2 | MCQ | What is the main risk source in this scene? A Road debris; B Cut-in vehicle; C Crossing pedestrian/cyclist; D Unclear lane boundary; E No clear immediate risk |
| SC-3 | MCQ | What is the preferred action for the ego vehicle? A Slow down; B Keep lane carefully; C Change lane; D Stop and wait; E Speed up |
| SC-4 | OEQ | Briefly describe the scene, the main risk, and the preferred action. |
| SP-3a | OEQ | What is the approximate distance in meters between <obj> and the ego vehicle? |
| TRJ-5 † | OEQ | Based on the past 5 observed front-view frames sampled over approximately 2.0 seconds, predict the next 5 future ego positions as a JSON array of 5 points. Current frame is . |
Autonomous Driving Benchmarks
Autonomous driving benchmarks evaluate how well vehicles perceive, predict, plan, and act safely across varied roads and conditions.
See all 140 papers