SAMoE-VLA: A Scene Adaptive Mixture-of-Experts Vision-Language-Action Model for Autonomous Driving
Authors: Zihan You, Hongwei Liu, Chenxu Dang, Zhe Wang, Sining Ang, Aoqi Wang, Yan Wang
Organizations: Institute for AI Industry Research (AIR), Tsinghua University · School of Instrument Science and Engineering, Southeast University · Zhili College, Tsinghua University · School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · Department of Automation, University of Science and Technology of China · Department of Automation, University of Science and Technology Beijing
Recent advances in Vision-Language-Action (VLA) models have shown promising capabilities in autonomous driving by leveraging the understanding and reasoning strengths of Large Language Models(LLMs).However, our empirical analysis reveals that directly applying existing token-level MoE mechanisms--which are inherited from LLM architectures--to VLA models results in unstable performance and safety degradation in autonomous driving, highlighting a misalignment between token-based expert specialization and scene-level decision-making.To address this, we propose SAMoE-VLA, a scene-adaptive Vision-Language-Action framework that conditions expert selection on structured scene representations instead of token embeddings. Our key idea is to derive the MoE routing signal from bird's-eye-view (BEV) features that encapsulates traffic scene context, enabling scenario-dependent expert weighting and merging tailored to distinct driving conditions. Furthermore, to support temporally consistent reasoning across world-knowledge, perception, language, and action, we introduce a Conditional Cross-Modal Causal Attention mechanism that integrates world state, linguistic intent, and action history into a unified causal reasoning process. Extensive experiments on the nuScenes open loop planning dataset and LangAuto closed-loop benchmark demonstrate that SAMoE-VLA achieves state-of-the-art performance, outperforming prior VLA-based and world-model-based approaches with fewer parameters.Our code will be released soon.
Figures & tables
Figure 1: Overview of our SAMoE-VLA. SAMoE-VLA employs two functional experts. A World-Language Expert : This module performs multimodal processing by integrating tokenized human instructions, Bird’s-Eye-View (BEV) tokens and soft prompts for world embeddings. A Planning Expert : This expert utilizes a structure based on a scene adaptive Mixture-of-Experts (SAMoE) layers routed by the scene representation extracted from Deformable Scene Encoder and receives ego-state tokens and noisy action tokens as its input. Our model unifies these experts through Conditional Cross-Modal Causal Attention(CMCA).
Figure 2: Overview of our Scene Adaptive MoE guided by Deformable Scene Encoder. SA-MoE is the layer of our proposed planning expert shown in figure 1 . BEV hidden is calculated only once during inference, while expert weights need to be calculated in every layer. Each layer has its own Linear head in Eq. (8) so weights vary per layer.
Method
L2 (m)
Collision (%)
1s
2s
3s
Avg
1s
2s
3s
Avg
Traditional End2End & World Model-based
Ego-MLP [ 27 ]
0.46
0.76
1.12
0.78
0.21
0.35
0.58
0.38
UniAD [ 14 ]
0.20
0.42
0.75
0.46
0.02
0.25
0.84
0.37
VAD [ 18 ]
0.17
0.34
0.60
0.37
0.04
0.27
0.67
0.33
Drive-WM [ 43 ]
0.43
0.77
1.20
0.80
0.10
0.21
0.48
0.26
Table 1: Comparison of planning performance on the nuScenes dataset. L2 Error is measured in meters and Collision Rate in percentage. Lower is better for both metrics. SAMoE-VLA reports total parameter count.
Method
LangAuto
LangAuto-S
LangAuto-T
DS
RC
IS
DS
RC
IS
DS
RC
IS
LMdrive(7B) [ 37 ]
36.2
46.5
0.81
50.6
60.0
0.84
66.5
77.9
0.85
AD-H(7B) [ 50 ]
44.0
53.2
0.83
56.1
68.0
0.78
77.5
85.1
0.91
DSDrive(1B) [ 30 ]
29.5
39.3
0.77
62.0
76.1
0.81
60.6
72.5
0.84
BEVDriver(7B) [ 44 ]
48.9
59.7
0.82
66.7
77.8
0.87
70.2
81.3
0.87
SAMoE-VLA (3.6B)
51.4
63.5
0.81
69.5
80.9
0.86
79.5
86.4
0.92
Table 2: Overall performance comparison on the close loop LangAuto benchmark under three evaluation settings:LangAuto, LangAuto-Short, and LangAuto-Tiny. Higher values indicate better performance.DS (Driving Score) reflects the overall driving quality by aggregating task completion and safety penalties; IS (Infraction Score) measures compliance by penalizing traffic violations; RC (Route Completion) quantifies the percentage of the assigned route successfully completed. SAMoE-VLA reports total parameter count.
Figure 5
Method
L2 ↓ (m)
Collision ↓ (%)
1s
2s
3s
Avg
1s
2s
3s
Avg
Prefix Routing
0.22
0.29
0.43
0.31
0.00
0.25
0.86
0.37
w/o DCN
0.26
0.28
0.37
0.31
0.02
0.21
0.76
0.33
w/o Mnear
0.25
0.26
0.38
0.30
0.01
0.20
0.66
0.29
Ours
0.24
0.27
0.35
0.29
0.01
0.18
0.60
0.26
Table 3: Comparison of L2 distance and collision rate for different routers.
Figure 7Figure 8
Figure 9: Comparison between different MoE
Method
L2 ↓ (m)
Collision ↓ (%)
1s
2s
3s
Avg
1s
2s
3s
Avg
w/o MOE
0.29
0.31
0.37
0.32
0.01
0.21
0.56
0.25
sparseMOE (w/o bev bias)
0.24
0.26
0.40
0.30
0.02
0.22
0.84
0.36
sparseMOE (with bev bias)
0.27
0.53
0.92
0.57
0.02
0.44
1.59
0.69
SoftMOE (w/o bev bias)
0.39
0.42
0.56
0.46
0.09
0.44
1.17
0.57
SoftMOE (with bev bias)
0.26
0.30
0.43
0.33
0.02
0.22
0.77
0.34
Table 4: Comparison of L2 distance and collision rate with different MoE designs.
x=[z,[MASK]state+action,w1:T]
Algorithm 3 Stage I(step3) — Pretraining of World–Language Expert
Δ=P([FBEV;Mnear]).
Algorithm 4 DSE-based Routing Preparation
FBEV=SCA(img_feats,BEV_queries,pos).
Algorithm 5 Final Training Stage of SAMoE-VLA
Subset
Samples
Avg Agent Count
Avg Yaw Rate (rad/s)
Avg Min Distance (m)
List 1 (high-density)
38
52.105263
0.076436
6.322976
List 2 (high-yaw)
27
46.259259
0.191537
12.770937
List 3 (close-prox.)
42
33.404762
0.014230
5.857343
nuScenes (full)
34,149
34.149960
0.037195
8.143113
Table 5: Summary statistics for the selected challenging subsets and the full nuScenes split. “Samples” denotes number of selected test cases.
Figure 10: At a signalized urban intersection, the ego vehicle is positioned in the left lane, where traffic is stationary while waiting for the red light. A concrete mixer truck occupies the adjacent right lane and is unable to change lanes. In this scenario, our model correctly keeps the ego vehicle stationary, closely matching the ground-truth trajectory.
Figure 11: This case locates at an urban four-way intersection. The ego vehicle approaches a left-turn–only lane and initiates a turn into a narrow lane bordered by roadside barriers, with many pedestrians nearby. Our model executes the left turn at a cautious speed, avoiding collisions with both the barriers and pedestrians, and its predicted trajectory closely matches the ground-truth path.
Figure 12: This case locates on a suburban two-lane road. The ego vehicle prepares to make a left turn on a narrow bidirectional roadway. In this scenario, our model successfully executes the left-turn maneuver, producing a predicted trajectory that closely matches the ground-truth path.
Figure 13: This case locates at a complex urban intersection. With a green traffic signal, the ego vehicle proceeds straight through the intersection while vehicles from multiple directions are either moving or waiting. In this challenging setting, our model successfully completes the straight-driving maneuver without any collisions, and its predicted trajectory closely follows the ground-truth path.
Figure 14: Case study of a lane-change maneuver near an intersection. The ego vehicle approaches a junction on a two-lane roadway, initially traveling in the left lane, which is blocked by a vehicle waiting at a red light. Two large vehicles are present behind in the adjacent right lane. In this scenario, the ego vehicle intends to merge into the right lane. Our model successfully performs the lane change, producing a predicted trajectory that closely matches the ground-truth path.
Figure 15: Case study at a large and complex urban intersection. With a green signal, the ego vehicle has passed the stop line and prepares to make a right turn into the adjacent roadway. Two nearby vehicles are present in the left lane, and additional traffic is approaching the intersection from other directions. In this scenario, our model generates a right-turn trajectory that closely follows the ground-truth path.
Figure 16: Case study at an urban intersection with a green signal. The ego vehicle proceeds straight through the intersection, where traffic is complex due to the presence of multiple vehicles and pedestrians, and the incoming road narrows to a single lane. We compare the trajectories generated by the MoE-enabled model and the non-MoE model. The MoE-enabled model produces a trajectory that closely matches the ground-truth path and correctly enters the target lane, whereas the non-MoE model deviates significantly and leads the vehicle into an invalid area.
Figure 17: Case study at a complex urban intersection with many vehicles and pedestrians. The ego vehicle prepares to make a right turn, with two nearby vehicles in close proximity. In this scenario, the MoE-enabled model produces a trajectory that closely follows the ground-truth path, while the non-MoE model exhibits a larger prediction error that increases over time.
Symbol
Description
θ
Model parameter vector in flow matching.
A∈{0,1}B×L×L
Binary attention mask enforcing CMCA.
C
Index set of conditioning tokens.
Sb,h,i,j
Original similarity score for head h , sequence b , query i , key j .
Sb,h,i,j′
Masked similarity score after CMCA eager-attention masking.
V
Concatenated value states from two experts.
Table 6: Notation table for all symbols used in the SAMoE-VLA framework.
Symbol
Description
B∈RH×W×Cb
BEV feature map extracted from multi-view images.
DSE:RCb→RD
Deformable Scene Encoder mapping BEV to D -dimensional.
HBEV∈R(HW)×D
Flattened BEV tokens after DSE encoding.
h , c , l
Ego-history, CAN bus readings, and low-level control features.
ψhis,ψbus,ψego
Linear projections for auxiliary context embeddings.
S∈RLs×D
Concatenated state/context tokens.
Table 7: Summary of Notation Used in Flow-Matching Implementation.
Symbol
Description
FBEV
BEV feature map.
DownSampleCross
Lightweight BEV downsampling module.
Etae
Reconstruction target for autoencoding regularization.
Nq
Number of world queries.
Q
Pooled BEV queries for LLM input.
fframe(k)
Learnable soft prompt for the k -th future frame.
Table 8: Notation used in implementation of World Modeling.
Integrating vision-language models (VLMs) into end-to-end (E2E) autonomous driving (AD) systems has shown promise in improving scene understanding. However, existing integration strategies suffer from several limitations: they either struggle to resolve distribution misalignment between reasoning and action spaces, underexploit the general reasoning capabilities of pretrained VLMs, or incur substantial inference latency during action policy generation, which degrades driving performance. To address these challenges, we propose AutoMoT in this work, an end-to-end AD framework that unifies reasoning and action generation within a single vision-language-action (VLA) model. Our approach leverages a mixture-of-transformer (MoT) architecture with joint attention sharing, which preserves the general reasoning capabilities of pre-trained VLMs while enabling efficient fast-slow inference through asynchronous execution at different task frequencies. Extensive experiments on multiple benchmarks, under both open- and closed-loop settings, demonstrate that AutoMoT achieves competitive performance compared to state-of-the-art methods. We further investigate the functional boundary of pre-trained VLMs in AD, examining when AD-tailored fine-tuning is necessary. Our results show that pre-trained VLMs can achieve competitive multi-task scene understanding performance through semantic prompting alone, while fine-tuning remains essential for action-level tasks such as decision-making and trajectory planning. We refer to https://automot-website.github.io/ for the demonstration videos and qualitative results.
Wenhui Huang, Songyan Zhang, Qihang Huang +6
Nanyang Technological University, Singapore · Harvard University, US · Xiaomi EV, China
Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.
Yuguang Yang, Canyu Chen, Zhewen Tan +10
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5
End-to-end autonomous driving requires models to understand traffic scenes, infer driving intent, and generate executable motion plans. Recent vision-language-action (VLA) models inherit semantic priors from large-scale vision-language pretraining, yet still face a coupling trade-off: fully shared backbones preserve multimodal interaction but may entangle language reasoning and trajectory prediction, whereas decou pled reasoning-action pipelines reduce task conflict but weaken semantic-motion coupling. We propose VECTOR-DRIVE, a tightly coupled VLA framework built on Qwen2.5-VL-3B. VECTOR-DRIVE keeps all tokens coupled through shared self attention and routes feed-forward computation according to token semantics. Vision and language tokens are processed by a Vision-Language Expert to preserve semantic priors, while target-point, ego-state, and noisy action tokens are routed to a Trajectory Expert for motion-specific computation. On the action-token pathway, a flow-matching planner refines noisy action tokens into future waypoints and speed profiles. This design couples semantic reasoning and motion planning within a single multimodal Transformer while separating task-specific FFN computation. On Bench2Drive, VECTOR-DRIVE achieves 88.91 Driving Score and outperforms representative end-to end and VLA-based baselines. Qualitative results and ablations further validate the benefits of shared attention, semantic-aware expert routing, progressive training, and flow-based action de coding.
Rui Zhao, Jianlin Yu, Zhenhai Gao +2
College of Automotive Engineering, Jilin University · 2The National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University · 3ReeFocus AI Technology