Organizations: Beihang University · Horizon Robotics · University of Macau · BAAI · AIR, Tsinghua University · OpenDriveLab, The University of Hong Kong
Real-world driving requires context-aware decision-making. Existing end-to-end driving benchmarks primarily assess trajectory quality through safety, compliance, comfort, and driving progress, leaving context-dependent decision-making insufficiently characterized. To fill this gap, we present PlanContext, a benchmark for evaluating in-context driving decision-making. PlanContext constructs paired evaluations by varying high-level driving intentions while keeping the observed scene fixed, enabling a natural and controllable assessment of context-dependent decision-making. We further define six representative driving contexts and jointly evaluate contextual decision correctness, road compliance, and dynamic safety with class-balanced metrics. Experiments on 9,561 scene-condition instances from NAVSIM and 23 public end-to-end driving models show that high conventional driving-quality scores do not consistently correspond to strong route-level in-context driving decision-making. Further analysis shows that model behavior varies with decision timing, road structure, and traffic interaction depending on the decision required by the current context. Our results highlight in-context driving decision-making as an important complementary dimension for evaluating end-to-end driving systems.
Figures & tables
Figure 1: PlanContext. Conventional driving metrics primarily evaluate trajectory quality in terms of safety, compliance, comfort, and driving progress. However, models with high PDMS can still produce context-inappropriate trajectories when navigation intentions change. PlanContext uses a natural and controllable navigation intervention to evaluate in-context driving decision-making.
Figure 2: Representative examples of the six PlanContext categories. Parentheses indicate the change in navigation intention.
Category
Context condition
Requirement
A
Target route reachable within 2 s
Follow
B1
Target route reachable in 4.5–8 s
Wait
B2
No locally feasible target
Reject
C
Ego beyond route commitment point
Reject
L
Adjacent target lane geometrically and topologically reachable
Follow
LC
Lateral maneuver already in progress
Preserve
Table 1: PlanContext scenario categories and contextual requirements.
Model
Venue
PDMS
mPDMS
NAS
CPS
NCS
LAS
MCS
VA Models
DrivoR ( Kirby et al., 2026 )
CVPR 2026
93.96
83.56
19.76
61.23
34.78
0.00
92.62
GoalFlow ( Xing et al., 2025 )
CVPR 2025
88.31
79.82
15.57
63.89
31.54
0.07
87.82
TransFuser ( Chitta et al., 2022 )
IEEE TPAMI
83.39
74.16
13.77
52.98
27.01
0.15
73.59
SparseDriveV2 ( Sun et al., 2026 )
ECCV 2026
92.22
81.22
12.57
56.25
26.60
0.15
93.64
DiffusionDrive ( Liao et al., 2025 )
CVPR 2025
88.05
79.80
3.29
65.44
14.68
0.00
92.11
Table 2: Overall comparison of 23 end-to-end driving models on PlanContext.
Figure 3: Driving quality and in-context performance across 23 end-to-end driving models.
Category
Instances
Success (%)
A
334
10.01
B1
251
62.91
B2
5,285
73.28
C
580
34.98
L
2,716
0.45
LC
395
85.51
Table 3: Contextual success across PlanContext categories.
Decision Urgency
Geometric Salience
Topological Evidence
Category
Low
Medium
High
Low
Medium
High
Δq [95% CI]
A
15.59
11.28
3.22
6.66
12.34
11.02
+13.15∗ [10.62, 15.72]
B1
64.22
61.90
62.63
55.74
77.43
55.49
+35.67∗ [21.22, 49.46]
B2
73.50
75.06
71.29
73.10
75.36
71.39
−13.45∗ [-15.16, -11.78]
C
23.74
43.39
37.79
29.96
36.09
38.86
−20.77∗ [-28.05, -13.39]
L
0.49
0.43
0.42
0.46
0.43
0.46
+0.08 [-0.07, 0.22]
Table 4: Associations between scene context factors and PlanContext performance.
Category
Lead Vehicle
TTC Risk
Target Occupancy
Crossing Conflict
Traffic Density
A
−4.88
−9.08∗
−9.94∗
−0.18
+5.31∗
B1
+17.47
−5.15
−6.00
+12.39
−0.88
B2
+9.05∗
+6.29∗
+5.54∗
−4.45∗
+12.10∗
C
+3.58
−10.36
−10.02∗
−11.51∗
−9.75∗
L
−0.50∗
−0.26∗
−0.39∗
+0.20
−0.13∗
LC
−7.62∗
−8.62∗
−6.88∗
−9.56∗
−7.82∗
Table 5: Associations between traffic interaction factors and PlanContext performance.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Instances
Description
Initial candidates
12,840
Original scene–condition candidates
After motion filtering
12,242
Remove near-stationary instances
Final benchmark
9,561
Apply final context-aware filtering
Appendix
Table 6: Scenario filtering statistics for PlanContext.
Category
Original condition
Alternative condition
A
Original branch
Target branch
B1
Original approach branch
Original approach branch
B2
Local valid roads and adjacent roads
Local valid roads and adjacent roads
C
Committed original branch
Committed original branch
L
Current lane
Current and target adjacent lanes
LC
Maneuver-related local roads
Maneuver-related local roads
Appendix
Table 7: Permitted road regions for the six PlanContext categories.
Category
Requirement
Instances
A
Follow
334
B1
Wait
251
B2
Reject
5,285
C
Reject
580
L
Follow
2,716
LC
Preserve
395
Appendix
Table 8: Composition of the PlanContext benchmark.
Model
NCS
95% CI
OneVL
39.30
[34.73, 43.41]
ChainFlow-VLA
35.13
[31.19, 38.83]
DrivoR
34.78
[30.76, 38.58]
ReCogDrive-IL
34.31
[30.38, 38.01]
SUV
32.03
[27.10, 36.55]
ReCogDrive-RL
31.64
[27.82, 35.18]
Appendix
Table 9: NCS and 95% scene-level bootstrap confidence intervals for the 23 evaluated models.
Setting
Minimum Spearman correlation
Maximum rank change
w∈[0.40,0.60]
0.9960
1
w∈[0.30,0.70]
0.9703
5
Omit B1 from CPS
0.9960
2
Omit B2 from CPS
0.9970
1
Omit C from CPS
0.9970
1
Appendix
Table 10: Robustness of the PlanContext ranking under metric variations.
Category
Alt. Correct.
RoadOK
Dynamic Safety
Overall Safety
Alt. Success
Contextual Success
A
44.94
19.49
66.12
11.95
11.44
10.01
B1
94.37
68.63
85.43
63.59
62.91
62.91
B2
92.50
79.86
84.19
73.95
73.28
73.28
C
88.12
46.87
67.93
35.85
35.01
34.98
L
3.51
97.57
94.16
93.55
1.58
0.45
LC
97.51
94.32
87.05
85.85
85.64
85.64
Appendix
Table 11: Atomic metric decomposition across the six PlanContext categories.
Figure 4: Category A: an immediately actionable alternative route. The navigation command changes from straight to left at an intersection where the alternative left-turn route is reachable within the evaluation horizon. The trajectories illustrate differences in how models respond to the alternative navigation intention.
Figure 5: Category B1: an alternative route beyond the current decision window. The navigation command changes from straight to right before the right-turn target becomes actionable. Continuing along the current road remains appropriate within the evaluation horizon. The trajectories illustrate differences between maintaining the current path and deviating prematurely toward the requested direction.
Figure 6: Category B2: a locally unsupported navigation intention. The navigation command changes from straight to right where the local road topology provides no feasible right-turn route. The trajectories illustrate differences between following the existing road geometry and deviating toward the unsupported direction.
Figure 7: Category C: a navigation intervention after route commitment. The navigation command changes from left to straight after the ego vehicle has passed the route commitment point. The trajectories illustrate differences between continuing along the committed route and deviating in response to the changed command. Collision annotations identify the marked failures in the displayed predictions.
Figure 8: Qualitative examples for OneVL. The first two cases show collisions following navigation interventions in categories C and A. The third case shows a directionally correct response in category C.
Figure 9: Qualitative examples for ChainFlow-VLA. The cases show an incorrect route choice after route commitment in category C, a collision in category B2, and a correct alternative route choice in category A.
Figure 10: Qualitative examples for DrivoR. The first case shows a collision in category B2. The two category A cases illustrate different outcomes when responding to an immediately actionable alternative navigation intention, including an incorrect route choice and a correct route choice.
Figure 11: Qualitative examples for StateMLP. The first two cases show collisions in categories B2 and L. The third case shows a correct alternative route choice in category A.
Figure 12: Qualitative examples for Curious-VLA. The first two cases show collisions in categories A and C. The third case shows continued travel along the committed route after the navigation command changes in category C.
Figure 13: Qualitative examples for CLOVER. The cases show an incorrect alternative route choice in category A, a collision in category B2, and a correct alternative route choice in another category A scene.
Figure 14: Qualitative examples for WAM-Flow. The two category C cases show a collision and travel against the traffic direction following navigation interventions after route commitment. The category A case shows a correct alternative route choice.
Figure 15: Qualitative examples for SimWAM. The first case shows an incorrect route choice in category C. The two category A cases show travel against the traffic direction and a correct alternative route choice, respectively.
Figure 16: Qualitative examples for Drive-JEPA. The cases show a collision in category L, little visible change between the paired predictions in category A, and travel against the traffic direction in category B2.
Figure 17: Qualitative examples for DynVLA Stage 2. The cases show little visible response to a lane-level navigation intervention in category L, an incorrect route choice in category A, and continued travel along the current road in category B2.
Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.
Autonomous driving has shifted towards end-to-end policy learning, where reliable, interpretable policy evaluation is a fundamental challenge as driving quality is highly context-dependent. Commonly used rule-based driving metrics like EPDMS are interpretable but lack context-awareness, while recent VLMbased evaluations are context-aware but limited by ambiguous VLM outputs and weak physical grounding. To evaluate driving in a manner that is both interpretable and context-aware, we introduce DriveJudge. DriveJudge is a driving evaluation agent that combines rule-grounded evaluation with Vision-Language Model (VLM) reasoning and selectively invokes physically-grounded deterministic rule functions after interpreting the environmental context. To train and evaluate DriveJudge, we curate a large-scale dataset of 33,577 challenging driving samples with human annotations on whether the driving behavior is reasonable in the given scenario. With this dataset, we address the underexplored problem of driving metric evaluation, and introduce two human-aligned benchmark tasks: Driving Quality Classification and Trajectory Preference Selection. DriveJudge outperforms EPDMS for driving quality classification by 21.23 AUC, and the recent VLM-based DriveCritic for trajectory preference selection by 6.5%, setting a new standard for interpretable and precise driving evaluation.
End-to-end autonomous driving models plan future trajectories from raw sensor input. While earlier driving benchmarks often measured deviation from the human trajectory, current benchmarks such as NAVSIM and Bench2Drive evaluate models with richer simulation-based metrics intended to capture safe and compliant driving. A high benchmark score should reflect that a model can understand the scene in front of it and act accordingly. But how much of that score specifically comes from reacting to the dynamic part of that scene? To probe this, we remove a model's camera input and replace it with memories from prior drives at the same location. The retrieved memories can provide persistent scene information, including road layout and location-conditioned regularities, but not the current traffic state. Surprisingly, memory is nearly sufficient on NAVSIM, reaching or even exceeding the performance of leading end-to-end methods without actually observing the evaluated scene. Our results suggest that a high NAVSIM score does not require a planner to react to the current traffic scene and should be treated with caution. This effect is benchmark-dependent: driving from memory causes substantially larger performance drops on Bench2Drive and RealEngine. We provide our code at https://github.com/boschresearch/MemoryDrivoR .
Christian Löwens, Thorben Funke, Alexandru Paul Condurache
Bosch Research · University of Lübeck · Automated Driving, Bosch
Closed-loop evaluation of end-to-end driving requires continuous rollouts that reveal how earlier decisions affect subsequent driving. However, existing benchmarks evaluate only short segments and fail to capture later consequences. Ambiguous directional commands also obscure the intended navigation objective. We introduce Odyssey, a closed-loop benchmark for long-horizon driving comprising 100 scenarios, each reconstructed from a 100-second nuPlan driving log to preserve the context of navigation maneuvers and traffic interactions. To provide a consistent navigation objective, Odyssey replaces directional commands with explicit standard-definition (SD) map routes that specify which roads to follow, while sensor-based planning determines local driving actions. Throughout these rollouts, diffusion-based refinement of 3DGS-rendered images reduces rendering artifacts along the ego trajectory. To assess how effectively planners follow these routes and prepare for upcoming maneuvers, we introduce SD Route Compliance and Pre-Lane Change Score. These assessments are complemented by RouteDS, which extends the Driving Score with penalties for SD-route deviations and failed lane preparation. We adapt state-of-the-art planners, including vision-language-action (VLA) models, and evaluate their navigation performance using these metrics. Odyssey highlights open questions in route representation and integration for E2E driving. Benchmark code and adapted baselines will be released publicly.