Integrating vision-language models (VLMs) into end-to-end (E2E) autonomous driving (AD) systems has shown promise in improving scene understanding. However, existing integration strategies suffer from several limitations: they either struggle to resolve distribution misalignment between reasoning and action spaces, underexploit the general reasoning capabilities of pretrained VLMs, or incur substantial inference latency during action policy generation, which degrades driving performance. To address these challenges, we propose AutoMoT in this work, an end-to-end AD framework that unifies reasoning and action generation within a single vision-language-action (VLA) model. Our approach leverages a mixture-of-transformer (MoT) architecture with joint attention sharing, which preserves the general reasoning capabilities of pre-trained VLMs while enabling efficient fast-slow inference through asynchronous execution at different task frequencies. Extensive experiments on multiple benchmarks, under both open- and closed-loop settings, demonstrate that AutoMoT achieves competitive performance compared to state-of-the-art methods. We further investigate the functional boundary of pre-trained VLMs in AD, examining when AD-tailored fine-tuning is necessary. Our results show that pre-trained VLMs can achieve competitive multi-task scene understanding performance through semantic prompting alone, while fine-tuning remains essential for action-level tasks such as decision-making and trajectory planning. We refer to https://automot-website.github.io/ for the demonstration videos and qualitative results.
Figures & tables
Figure 1: Comparison of different paradigms for integrating VLMs into conventional end-to-end autonomous driving frameworks. Our AutoMoT framework unifies reasoning and action-policy learning within a single vision–language–action (VLA) model through layer-wise joint attention sharing, while enabling asynchronous inference across tasks with different execution frequencies.
Figure 2: As an end-to-end autonomous driving framework, AutoMoT unifies scene understanding, decision-making, and trajectory planning within a single VLA model. AutoMoT adopts a MoT architecture that connects the understanding expert and the action expert via layer-wise joint attention sharing, while enabling asynchronous inference at different frequencies through KV Caching. A VLA-oriented action refiner is further explored to enhance driving performance via diffusion-based refinement.
Figure 3: Our mask coordinates understanding, decision-making, and planning within a unified attention space. It enables intra-task multi-modal aggregation and cross-task information flow while preserving task-level causal ordering. This hybrid design maintains hierarchical causality and supports rich contextual integration, enabling AutoMoT to achieve coherent multi-task reasoning, decision-making, and trajectory planning.
Figure 4: The architecture of DiT-based action refiner with Mixture-of-Attention blocks.
Method
Expert
VLM
Generative
Closed-loop Metric
Planner
DS ↑
SR(%) ↑
MomAD ( Song et al., 2025 )
Think2Drive
-
-
44.54
16.71
UniAD-Base ( Hu et al., 2023 )
Think2Drive
-
-
45.81
16.36
TCP-traj ( Wu et al., 2022 )
Think2Drive
-
-
59.90
30.00
DriveTransformer-Large ( Jia et al., 2025 )
Think2Drive
-
-
63.46
35.01
DriveAdapter ( Jia et al., 2023 )
Think2Drive
-
-
64.22
33.08
Table 1: Comparison of closed-loop planning performance on the CARLA Bench2Drive leaderboard. C/L denotes camera/LiDAR input. DS and SR represent Driving Score and Success Rate, respectively.
Method
Ego Status
Finetuning
L2 (m) ↓
Collision (%) ↓
Understanding
Decision
Planning
1s
2s
3s
Avg.
1s
2s
3s
Avg.
UniAD ( Hu et al., 2023 )
Vector
-
-
✓
0.44
0.67
0.96
0.69
0.04
0.08
0.23
0.12
VAD ( Jiang et al., 2023 )
Vector
-
-
✓
0.17
0.34
0.60
0.37
0.07
0.10
0.24
0.14
Ego-MLP ( Li et al., 2024 )
Vector
-
-
✓
0.15
0.32
0.59
0.35
0.00
0.27
0.85
0.37
DriveTransformer-Large ( Jia et al., 2025 )
Vector
-
-
✓
0.16
0.30
0.55
0.33
0.01
0.06
0.15
0.07
AutoVLA ( Zhou et al., 2025c )
Text
✓
-
✓
0.21
0.38
0.60
0.40
0.13
0.18
0.28
0.20
Table 2: Comparison of the Open-loop planning in nuScenes. The ST-P3 evaluation protocol is used by default.
Method
LingoQA
OmniDrive
CODA-LM
TallyQA
InfoVQA
ReCogDrive
67.20
0.82
5.90
69.60
75.80
Robotron-Drive †
59.20
0.82
6.20
63.40
42.60
OpenEMMA
48.00
0.43
4.80
80.00
71.40
AutoMoT
67.00
0.89
6.07
81.40
89.30
Table 3: Comparison of reasoning capabilities across both general-domain and autonomous driving–specific datasets. † : Results are reproduced using the official checkpoints and evaluation environments.
Benchmark
Task Category
AutoMoT †
AutoMoT ‡
LingoQA (L)
Scene Understanding
67.00
67.20
OmniDrive (G)
Counterfactual Planning
18.20
67.80
ScienceQA (A)
General Knowledge
88.60
87.80
FigureQA (A)
General Knowledge
97.60
91.20
TallyQA (A)
General Knowledge
81.40
52.40
InfographicVQA (G)
General Knowledge
89.30
50.20
Table 4: Ablation study results of investigating the performance boundary of the pre-trained backbone. † : System prompt is provided; ‡ : Fine-tuned on autonomous driving datasets; L: Lingo-Judge; G:GPT-Score; A: Token Accuracy.
Setting
L2@1s ↓
L2@2s ↓
L2@3s ↓
L2 avg↓
AutoMoT-S
0.140
0.290
0.537
0.322
AutoMoT
0.141
0.293
0.544
0.324
Table 5: Trajectory planning performance under synchronized and asynchronous settings. AutoMoT refers to the proposed model with the KV cache enabled, while AutoMoT-S denotes the synchronized variant that runs the understanding expert (UE) and action expert (AE) at the same frequency, without introducing temporal misalignment between UE and AE.
Setting
Generative
Time (ms)
Frequency
Planner
UE
AE
AR
Total
Hz
AutoMoT-S
-
80.3
37.0
-
117.3
8.5
AutoMoT
-
0.0
37.0
-
37.0
27.0
AutoMoT-S
✓
80.3
37.0
26.0
143.3
7.0
AutoMoT
✓
0.0
37.0
26.0
63.0
16.0
Table 6: Latency breakdown of AutoMoT under synchronized (AutoMoT-S) and default AutoMoT settings.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: The detailed data flow of AutoMoT.
Lateral Acc. ↑
Longitudinal Acc. ↑
Joint Acc. ↑
Method
1s
2s
Avg
1s
2s
Avg
1s
2s
Avg
AutoMoT-S
95.00%
83.20%
84.50%
77.38%
56.81%
62.28%
73.84%
46.77%
53.49%
AutoMoT
94.06%
82.69%
83.79%
77.57%
56.85%
62.38%
73.40%
46.36%
53.10%
Appendix
Table 7: Decision-making accuracy under synchronized and asynchronous settings at different time horizons.
Model
Fine-tuned
Accuracy (%)
Senna
✓
88.47
AutoMoT (Ours)
✓
90.92
Appendix
Table 8: Decision-making performance comparison on the Senna nuScenes benchmark.
Method
L2@1s ↓
L2@2s ↓
L2@3s ↓
L2 avg ↓
AutoMoT (Ours)
0.14
0.29
0.54
0.32
AutoMoT-R (w/o Pre-trained UE)
0.16
0.33
0.60
0.36
AutoMoT-P (w/o decision-making)
0.14
0.30
0.58
0.34
Appendix
Table 9: Ablation study on the understanding and decision-making components of AutoMoT. AutoMoT-R denotes the variant with a randomly initialized VLM backbone. AutoMoT-P denotes the planning-only variant, where the action expert is trained without the decision-making (meta-action planning) objective.
Method
GPU
Latency (ms)
Hz
OpenEMMA
RTX 5090
7,683
0.13
AutoVLA † (fast)
Unknown
1,072
0.93
AutoVLA † (slow)
Unknown
10,518
0.10
SimLingo
RTX 5090
430
2.3
AutoMoT-S (sync)
RTX 5090
117
8.5
AutoMoT (async)
RTX 5090
37
27
Appendix
Table 10: Inference latency comparison of VLA-based driving methods. † Results are cited from the original paper, as public checkpoints are not available for reproduction. The inference GPU is not specified; training uses 8× NVIDIA L40S.
Institute for AI Industry Research (AIR), Tsinghua University · School of Instrument Science and Engineering, Southeast University · Zhili College, Tsinghua University +3
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University +2
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA