SAMoE-VLA: A Scene Adaptive Mixture-of-Experts Vision-Language-Action Model for Autonomous Driving

Zihan You; Hongwei Liu; Chenxu Dang; Zhe Wang; Sining Ang; Aoqi Wang; Yan Wang

cs.CVMar 9, 2026

SAMoE-VLA: A Scene Adaptive Mixture-of-Experts Vision-Language-Action Model for Autonomous Driving

Authors: Zihan You, Hongwei Liu, Chenxu Dang, Zhe Wang, Sining Ang, Aoqi Wang, Yan Wang

Organizations: Institute for AI Industry Research (AIR), Tsinghua University · School of Instrument Science and Engineering, Southeast University · Zhili College, Tsinghua University · School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · Department of Automation, University of Science and Technology of China · Department of Automation, University of Science and Technology Beijing

Abstract

Recent advances in Vision-Language-Action (VLA) models have shown promising capabilities in autonomous driving by leveraging the understanding and reasoning strengths of Large Language Models(LLMs).However, our empirical analysis reveals that directly applying existing token-level MoE mechanisms--which are inherited from LLM architectures--to VLA models results in unstable performance and safety degradation in autonomous driving, highlighting a misalignment between token-based expert specialization and scene-level decision-making.To address this, we propose SAMoE-VLA, a scene-adaptive Vision-Language-Action framework that conditions expert selection on structured scene representations instead of token embeddings. Our key idea is to derive the MoE routing signal from bird's-eye-view (BEV) features that encapsulates traffic scene context, enabling scenario-dependent expert weighting and merging tailored to distinct driving conditions. Furthermore, to support temporally consistent reasoning across world-knowledge, perception, language, and action, we introduce a Conditional Cross-Modal Causal Attention mechanism that integrates world state, linguistic intent, and action history into a unified causal reasoning process. Extensive experiments on the nuScenes open loop planning dataset and LangAuto closed-loop benchmark demonstrate that SAMoE-VLA achieves state-of-the-art performance, outperforming prior VLA-based and world-model-based approaches with fewer parameters.Our code will be released soon.

Figures & tables

Figure 1: Overview of our SAMoE-VLA. SAMoE-VLA employs two functional experts. A World-Language Expert : This module performs multimodal processing by integrating tokenized human instructions, Bird’s-Eye-View (BEV) tokens and soft prompts for world embeddings. A Planning Expert : This expert utilizes a structure based on a scene adaptive Mixture-of-Experts (SAMoE) layers routed by the scene representation extracted from Deformable Scene Encoder and receives ego-state tokens and noisy action tokens as its input. Our model unifies these experts through Conditional Cross-Modal Causal Attention(CMCA).

Figure 2: Overview of our Scene Adaptive MoE guided by Deformable Scene Encoder. SA-MoE is the layer of our proposed planning expert shown in figure 1 . BEV hidden is calculated only once during inference, while expert weights need to be calculated in every layer. Each layer has its own Linear head in Eq. (8) so weights vary per layer.

Traditional End2End & World Model-based
Method	L2 (m)				Collision (%)
	1s	2s	3s	Avg	1s	2s	3s	Avg
Ego-MLP [ 27 ]	0.46	0.76	1.12	0.78	0.21	0.35	0.58	0.38
UniAD [ 14 ]	0.20	0.42	0.75	0.46	0.02	0.25	0.84	0.37
VAD [ 18 ]	0.17	0.34	0.60	0.37	0.04	0.27	0.67	0.33
Drive-WM [ 43 ]	0.43	0.77	1.20	0.80	0.10	0.21	0.48	0.26

Table 1: Comparison of planning performance on the nuScenes dataset. L2 Error is measured in meters and Collision Rate in percentage. Lower is better for both metrics. SAMoE-VLA reports total parameter count.

Method	LangAuto			LangAuto-S			LangAuto-T
	DS	RC	IS	DS	RC	IS	DS	RC	IS
LMdrive(7B) [ 37 ]	36.2	46.5	0.81	50.6	60.0	0.84	66.5	77.9	0.85
AD-H(7B) [ 50 ]	44.0	53.2	0.83	56.1	68.0	0.78	77.5	85.1	0.91
DSDrive(1B) [ 30 ]	29.5	39.3	0.77	62.0	76.1	0.81	60.6	72.5	0.84
BEVDriver(7B) [ 44 ]	48.9	59.7	0.82	66.7	77.8	0.87	70.2	81.3	0.87
SAMoE-VLA (3.6B)	51.4	63.5	0.81	69.5	80.9	0.86	79.5	86.4	0.92

Table 2: Overall performance comparison on the close loop LangAuto benchmark under three evaluation settings:LangAuto, LangAuto-Short, and LangAuto-Tiny. Higher values indicate better performance.DS (Driving Score) reflects the overall driving quality by aggregating task completion and safety penalties; IS (Infraction Score) measures compliance by penalizing traffic violations; RC (Route Completion) quantifies the percentage of the assigned route successfully completed. SAMoE-VLA reports total parameter count.

Method	L2 $\downarrow$ (m)				Collision $\downarrow$ (%)
	1s	2s	3s	Avg	1s	2s	3s	Avg
Prefix Routing	0.22	0.29	0.43	0.31	0.00	0.25	0.86	0.37
w/o DCN	0.26	0.28	0.37	0.31	0.02	0.21	0.76	0.33
w/o $\mathbf{M_{near}}$	0.25	0.26	0.38	0.30	0.01	0.20	0.66	0.29
Ours	0.24	0.27	0.35	0.29	0.01	0.18	0.60	0.26

Table 3: Comparison of L2 distance and collision rate for different routers.

Figure 9: Comparison between different MoE

Method	L2 $\downarrow$ (m)				Collision $\downarrow$ (%)
	1s	2s	3s	Avg	1s	2s	3s	Avg
w/o MOE	0.29	0.31	0.37	0.32	0.01	0.21	0.56	0.25
sparseMOE (w/o bev bias)	0.24	0.26	0.40	0.30	0.02	0.22	0.84	0.36
sparseMOE (with bev bias)	0.27	0.53	0.92	0.57	0.02	0.44	1.59	0.69
SoftMOE (w/o bev bias)	0.39	0.42	0.56	0.46	0.09	0.44	1.17	0.57
SoftMOE (with bev bias)	0.26	0.30	0.43	0.33	0.02	0.22	0.77	0.34

Table 4: Comparison of L2 distance and collision rate with different MoE designs.

\mathbf{x}=[\mathbf{z},\,\text{[MASK]}_{\text{state+action}},\,w_{1:T}]

Algorithm 3 Stage I(step3) — Pretraining of World–Language Expert

\Delta=\mathcal{P}\!\left([\mathbf{F}_{\mathrm{BEV}};M_{\mathrm{near}}]\right).

Algorithm 4 DSE-based Routing Preparation

\mathbf{F}_{\mathrm{BEV}}=\mathrm{SCA}(\text{img\_feats},\text{BEV\_queries},\text{pos}).

Algorithm 5 Final Training Stage of SAMoE-VLA

Subset	Samples	Avg Agent Count	Avg Yaw Rate (rad/s)	Avg Min Distance (m)
List 1 (high-density)	38	52.105263	0.076436	6.322976
List 2 (high-yaw)	27	46.259259	0.191537	12.770937
List 3 (close-prox.)	42	33.404762	0.014230	5.857343
nuScenes (full)	34,149	34.149960	0.037195	8.143113

Table 5: Summary statistics for the selected challenging subsets and the full nuScenes split. “Samples” denotes number of selected test cases.

Figure 10: At a signalized urban intersection, the ego vehicle is positioned in the left lane, where traffic is stationary while waiting for the red light. A concrete mixer truck occupies the adjacent right lane and is unable to change lanes. In this scenario, our model correctly keeps the ego vehicle stationary, closely matching the ground-truth trajectory.

Figure 11: This case locates at an urban four-way intersection. The ego vehicle approaches a left-turn–only lane and initiates a turn into a narrow lane bordered by roadside barriers, with many pedestrians nearby. Our model executes the left turn at a cautious speed, avoiding collisions with both the barriers and pedestrians, and its predicted trajectory closely matches the ground-truth path.

Figure 12: This case locates on a suburban two-lane road. The ego vehicle prepares to make a left turn on a narrow bidirectional roadway. In this scenario, our model successfully executes the left-turn maneuver, producing a predicted trajectory that closely matches the ground-truth path.

Figure 13: This case locates at a complex urban intersection. With a green traffic signal, the ego vehicle proceeds straight through the intersection while vehicles from multiple directions are either moving or waiting. In this challenging setting, our model successfully completes the straight-driving maneuver without any collisions, and its predicted trajectory closely follows the ground-truth path.

Figure 14: Case study of a lane-change maneuver near an intersection. The ego vehicle approaches a junction on a two-lane roadway, initially traveling in the left lane, which is blocked by a vehicle waiting at a red light. Two large vehicles are present behind in the adjacent right lane. In this scenario, the ego vehicle intends to merge into the right lane. Our model successfully performs the lane change, producing a predicted trajectory that closely matches the ground-truth path.

Figure 15: Case study at a large and complex urban intersection. With a green signal, the ego vehicle has passed the stop line and prepares to make a right turn into the adjacent roadway. Two nearby vehicles are present in the left lane, and additional traffic is approaching the intersection from other directions. In this scenario, our model generates a right-turn trajectory that closely follows the ground-truth path.

Figure 16: Case study at an urban intersection with a green signal. The ego vehicle proceeds straight through the intersection, where traffic is complex due to the presence of multiple vehicles and pedestrians, and the incoming road narrows to a single lane. We compare the trajectories generated by the MoE-enabled model and the non-MoE model. The MoE-enabled model produces a trajectory that closely matches the ground-truth path and correctly enters the target lane, whereas the non-MoE model deviates significantly and leads the vehicle into an invalid area.

Figure 17: Case study at a complex urban intersection with many vehicles and pedestrians. The ego vehicle prepares to make a right turn, with two nearby vehicles in close proximity. In this scenario, the MoE-enabled model produces a trajectory that closely follows the ground-truth path, while the non-MoE model exhibits a larger prediction error that increases over time.

Symbol	Description
$\theta$	Model parameter vector in flow matching.
$\mathbf{A}\in\{0,1\}^{B\times L\times L}$	Binary attention mask enforcing CMCA.
$\mathcal{C}$	Index set of conditioning tokens.
$\mathbf{S}_{b,h,i,j}$	Original similarity score for head $h$ , sequence $b$ , query $i$ , key $j$ .
$\mathbf{S}^{\prime}_{b,h,i,j}$	Masked similarity score after CMCA eager-attention masking.
$\mathbf{V}$	Concatenated value states from two experts.

Table 6: Notation table for all symbols used in the SAMoE-VLA framework.

Symbol	Description
$\mathbf{B}\in\mathbb{R}^{H\times W\times C_{b}}$	BEV feature map extracted from multi-view images.
$\mathbf{DSE}:\mathbb{R}^{C_{b}}\to\mathbb{R}^{D}$	Deformable Scene Encoder mapping BEV to $D$ -dimensional.
$\mathbf{H}_{\mathrm{BEV}}\in\mathbb{R}^{(HW)\times D}$	Flattened BEV tokens after DSE encoding.
$\mathbf{h}$ , $\mathbf{c}$ , $\mathbf{l}$	Ego-history, CAN bus readings, and low-level control features.
$\psi_{\mathrm{his}},\psi_{\mathrm{bus}},\psi_{\mathrm{ego}}$	Linear projections for auxiliary context embeddings.
$\mathbf{S}\in\mathbb{R}^{L_{s}\times D}$	Concatenated state/context tokens.

Table 7: Summary of Notation Used in Flow-Matching Implementation.

Symbol	Description
$\mathbf{F}_{BEV}$	BEV feature map.
$\mathrm{DownSampleCross}$	Lightweight BEV downsampling module.
$\mathbf{E}^{\mathrm{ae}}_{t}$	Reconstruction target for autoencoding regularization.
$N_{q}$	Number of world queries.
$\mathbf{Q}$	Pooled BEV queries for LLM input.
$\mathbf{f}_{\mathrm{frame}}(k)$	Learnable soft prompt for the $k$ -th future frame.

Table 8: Notation used in implementation of World Modeling.

Explore similar work

AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Driving
Mar 16, 2026Wenhui Huang, Songyan Zhang, Qihang Huang +6End-to-End Autonomous Driving Vision-Language-Action Models
Nanyang Technological University, Singapore · Harvard University, US · Xiaomi EV, China
Teaching Vision-Language-Action Models What to See and Where to Look
Jul 2, 2026Yuguang Yang, Canyu Chen, Zhewen Tan +10End-to-End Autonomous Driving VLM Distillation
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5
VECTOR-Drive: Tightly Coupled Vision-Language and Trajectory Expert Routing for End-to-End Autonomous Driving
May 9, 2026Rui Zhao, Jianlin Yu, Zhenhai Gao +2VLMs for Autonomous Driving End-to-End Autonomous Driving
College of Automotive Engineering, Jilin University · 2The National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University · 3ReeFocus AI Technology

Mar 16, 2026J/K move · Enter open · S save

AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Driving

Wenhui Huang, Songyan Zhang, Qihang Huang, Zhidong Wang+5

Nanyang Technological University, Singapore · Harvard University, US · Xiaomi EV, China

End-to-End Autonomous Driving Vision-Language-Action Models VLMs for Autonomous Driving Efficient VLA Model Inference Reasoning for VLA Models

Open paper AskPDF arXiv

Integrating vision-language models (VLMs) into end-to-end (E2E) autonomous driving (AD) systems has shown promise in improving scene understanding. However, existing integration strategies suffer from several limitations: they either struggle to resolve distribution misalignment between reasoning and action spaces, underexploit the general reasoning capabilities of pretrained VLMs, or incur substantial inference latency during action policy generation, which degrades driving performance. To address these challenges, we propose AutoMoT in this work, an end-to-end AD framework that unifies reasoning and action generation within a single vision-language-action (VLA) model. Our approach leverages a mixture-of-transformer (MoT) architecture with joint attention sharing, which preserves the general reasoning capabilities of pre-trained VLMs while enabling efficient fast-slow inference through asynchronous execution at different task frequencies. Extensive experiments on multiple benchmarks, under both open- and closed-loop settings, demonstrate that AutoMoT achieves competitive performance compared to state-of-the-art methods. We further investigate the functional boundary of pre-trained VLMs in AD, examining when AD-tailored fine-tuning is necessary. Our results show that pre-trained VLMs can achieve competitive multi-task scene understanding performance through semantic prompting alone, while fine-tuning remains essential for action-level tasks such as decision-making and trajectory planning. We refer to https://automot-website.github.io/ for the demonstration videos and qualitative results.

Abstract

Explore similar work

AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Driving

Teaching Vision-Language-Action Models What to See and Where to Look

VECTOR-Drive: Tightly Coupled Vision-Language and Trajectory Expert Routing for End-to-End Autonomous Driving