Organizations: College of Automotive and Energy Engineering, Tongji University, Shanghai, 201804, China · Tongji Automotive Design & Research Institute Co., Ltd., Shanghai, 201804, China · Hubei Jingchu Humanoid Robot Co., Ltd., Wuhan, Hubei, China · University of Chicago, Chicago, IL, 60637, USA
Vision-language-action (VLA) models for autonomous driving jointly produce scene interpretation, language-based reasoning, and driving trajectories. Existing evaluations often use independently selected synthetic, simulated, and physical data, so measured performance gaps can be confounded by changes in scenario content rather than genuine domain sensitivity. We propose SSP (Synthetic-Simulation-Physical), an event-matched Syn2Sim2Phy evaluation framework that anchors cross-domain comparison to the same safety-critical interaction. Starting from a synthetic long-tail video, SSP builds a validated event specification that preserves road topology, participant roles, relative motion, conflict evolution, passing order, response constraints, and event phases. Platform-specific realizations are then constructed in CARLA and on a closed proving ground and are evaluated only after transfer audits confirm preservation of mandatory event properties. SSP maps heterogeneous outputs from OpenEMMA, LLaViDA, and Alpamayo-R1 into common semantic slots and a 1 s trajectory window to assess output validity, semantic accuracy, critical-interaction recognition, trajectory quality, and risk response. Across Cut-in and vulnerable-road-user crossing cases, the macro-averaged Integrated VLA Capability Scores are 0.259, 0.291, and 0.325 in the Synthetic, Simulation, and Physical domains, respectively, while the best domain varies by scenario. Alpamayo-R1, OpenEMMA, and LLaViDA obtain scores of 0.405, 0.338, and 0.131. SSP provides a reproducible scene-transfer chain and an evidence-qualified evaluation of VLA behavior without assuming that the Physical domain is universally superior.
Figures & tables
Figure 1: Conceptual framework of SSP for event-matched Syn2Sim2Phy cross-domain evaluation of autonomous-driving VLA models.
Figure 2: Overall SSP workflow from synthetic source scenes to physical deployment and unified VLA evaluation.
Figure 3: Event-matched Syn2Sim2Phy transfer examples for Cut-in and VRU-crossing events. Colored conflict-region annotations and other overlays are used only for visualization and are not provided as inputs to the VLA models.
Figure 4: Projected Alpamayo-R1 inference results for the VRU-crossing event across the Synthetic, Simulation, and Physical domains. Predicted trajectories and action responses are overlaid on phase-aligned forward-view frames for qualitative cross-domain comparison.
Figure 5: Projected Alpamayo-R1 inference results for the Cut-in event across the Synthetic, Simulation, and Physical domains. Predicted trajectories and action responses are overlaid on phase-aligned forward-view frames for qualitative cross-domain comparison.
Figure 6: (a) Capability-wise evaluation matrix across two scenarios, three domains, and three VLA paradigms. (b) Cell-level IVCS, risk-response outcomes, and response latency across scenario–domain–model combinations.
Domain
JOVR
SemAcceff
CIAeff
TQSR
RSR
IVCS
Rank
Physical
0.926
0.222
0.021
0.370
0.500
0.325
1
Simulation
0.889
0.216
0.021
0.389
0.167
0.291
2
Synthetic
0.833
0.122
0.000
0.352
0.333
0.259
3
Table 2: Macro-averaged results by domain.
Figure 7: (a) Macro-averaged capability comparison across the three domains. (b) Domain-rank migration across Cut-in, macro average, and VRU. (c) Weighted IVCS contribution decomposition across domains.
Model
JOVR
SemAcceff
CIAeff
TQSR
RSR
IVCS
Rank
Alpamayo-R1
1.000
0.324
0.042
0.555
0.333
0.405
1
OpenEMMA
1.000
0.237
0.000
0.444
0.333
0.338
2
LLaViDA
0.648
0.000
0.000
0.111
0.333
0.131
3
Table 3: Macro-averaged results by VLA paradigm.
Figure 8: (a) Scenario-stratified and macro-averaged IVCS of the three VLA paradigms. (b) Capability-chain profiles of the three VLA paradigms. (c) Metric-wise performance differences of LLaViDA and Alpamayo-R1 relative to OpenEMMA.
Figure 9: (a) Performance trade-off between semantic accuracy and trajectory quality. (b) Relationship between active parameter count and TQSR for Qwen3 variants. (c) Capability trade-offs between Qwen3-VL-32B Dense and 30B-A3B MoE. (d) IVCS ranking of five OpenEMMA-style models with Alpamayo-R1.
Vision-language-action (VLA) models generate chain-of-thought (CoT) reasoning alongside driving trajectories, but existing benchmarks evaluate only trajectory quality and do not assess whether the CoT is relevant, consistent, or causally connected to the driving action. We introduce VLADriveBench, a framework that combines observational metrics (mentioning, hallucination, contradiction, action alignment) with a CoT intervention protocol to provide complementary views of the CoT-action relationship. Applying VLADriveBench to three models across two architectures, we find that the two analyses can diverge sharply: ORION scores highest on observational alignment yet its CoT is epiphenomenal, while Alpamayo v1.5 scores lower yet its CoT is strongly causal, with visual salience gating the extent of CoT influence.
Vision-Language-Action (VLA) models are gaining attention in autonomous driving, inheriting commonsense priors from large language models (LLMs) and widening the navigation interface from a fixed command set to free-form natural-language instructions. However, this open-ended interface exposes a new vulnerability surface: an inattentive driver may issue instructions that are unsafe given the current scene, and a malicious user may issue dangerous ones on purpose. We present the first benchmark suite that evaluates navigation following and conflict awareness together, along with adversarial robustness. Existing VLAs fall short on both: SimLingo completes no language-instructed lane change in closed loop, and all models degrade sharply under conflicting signals. We propose SafeDriveVLA, the first driving VLA that (1) explicitly predicts a three-level navigation-compliance mode to decouple conflict reasoning from action generation, and (2) uses navigation-conditioned world model dreaming to inform this prediction. Unlike prior world models that roll out expert or planned actions, ours dreams of strictly following the navigation signal, showing when following is safe. SafeDriveVLA reaches 88.16 Driving Score (DS) and 68.64% Success Rate (SR) on Bench2Drive, lifts closed-loop language following by 49.3% and reduces collisions under unsafe instructions from 60 to 38 compared to the best SimLingo variant. Video demonstrations are available on our project page.
Shaoyuan Xie, Zihan Zhang, Jingxuan Wang +6
UC Irvine · UC San Diego · University of Cincinnati +1
Vision-Language-Action (VLA) models for autonomous driving receive navigation commands and hazard notices as disconnected text fragments, forcing the model to discover on its own which environmental constraints are relevant to the current maneuver. We introduce Causal Scene Narration (CSN), which restructures VLA text inputs through intent-constraint alignment, quantitative grounding, and structured separation at inference time with zero GPU cost. To test CSN across different model states and safety layers, we evaluate it alongside training-time preference alignment (PL-DPO-NLL) and runtime supervision as experimental conditions. A multi-town CARLA evaluation (16 routes, 8 towns, N=5, 95% bootstrap CIs) shows that CSN yields relative improvements in CARLA Driving Score (DS, route completion weighted by infraction penalty) of +31.1% on original LMDrive and +24.5% on the PL-DPO-NLL variant. A controlled ablation separating information content from text organization reveals that information content carries most of CSN's gain across both model weights (+6.17 DS on original LMDrive; +6.88 DS on PL-DPO-NLL). While causal structure accounts for an estimated 39.1% (95% CI [6.6%, 71.4%]) of the gain on original LMDrive versus 13.5% on PL-DPO-NLL, the structural share on the aligned model and the cross-configuration asymmetry remain statistically inconclusive at N=5 due to wide bootstrap intervals. A perception noise ablation further confirms that CSN's benefit survives distance errors up to +/-5 m, speed noise up to +/-30%, and 20% actor miss rates. Finally, we report a negative result on combining CSN with runtime supervision: a naive direction-conflict monitor never fires on this benchmark and, when paired with CSN, degrades DS via passive control clamping that truncates CSN-guided evasive maneuvers. Code is available at https://github.com/LiYun0607/CausalSceneNarration.
Yun Li, Yidu Zhang, Simon Thompson +2
Graduate School of Information Science and Technology, The University of Tokyo · TIER IV, Inc.