A Collaborative Multi-Modality Interaction for VLA-based End-to-End Autonomous Driving

Jingtao Sun; Xiaohai He; Yike Zhang; Dong Huang; Yaonan Wang; Ajmal Mian; Mike Zheng Shou

cs.CVAug 21, 2026

A Collaborative Multi-Modality Interaction for VLA-based End-to-End Autonomous Driving

Authors: Jingtao Sun, Xiaohai He, Yike Zhang, Dong Huang, Yaonan Wang, Ajmal Mian, Mike Zheng Shou

Abstract

Vision-Language-Action (VLA) models have emerged as a powerful paradigm for end-to-end autonomous driving by jointly integrating perception, reasoning, and decision making within a unified multimodal framework. However, most existing VLA models formulate end-to-end autonomous driving as a visual question answering task, leading to unreliable and less interpretable decision reasoning. In addition, they fail to establish effective multi-modal interaction across heterogeneous sensors, thereby limiting robust scene perception and reliable driving reasoning in long-tail driving scenarios. To this end, we propose a robust VLA-based end-to-end autonomous driving system that combines multi-modality interaction with multi-trajectory planning and optimization, enabling more reliable, interpretable, and safer driving decisions. Our method comprises three core components: (1) Affinity-Guided Optimal Transport for main-auxiliary modality two-way interaction; (2) Distribution-Consistent Modality Transfer for heterogeneous modality distribution transfer and cross-modal interaction; (3) Multi-modal Multi-Trajectory Planning along with Perception-Oriented Trajectory Refinement for better driving decisions to long-tail driving scenarios. Experimental results in open-loop and closed-loop datasets demonstrate improvements in safety long-horizon driving reasoning and road scene perception over existing driving systems, highlighting the ability of our mutli-modality interaction and multi-trajectory planning and optimization for scalable VLA-based systems.

Figures & tables

Fig. 1: Comparison of different E2E-AD systems. (a) Existing fusion & interaction methods for E2E-AD normally fuse individual per-modality into a single hybrid BEV cue. (b) VLM or VLA model focuses more on answer question manner, they lack interpretability and reliable control decisions. (c) Our method achieves multi-modality interaction via optimal transport and generates optimized tarjectory, enhancing the driver trust in E2E-AD.

Fig. 2: Architecture of the proposed multi-modality interaction framework. Our method takes as input a front-view image, LiDAR point cloud, and a natural-language navigation command with a system prompt, and outputs the fused visual, BEV and text tokens. ”Affinity-guided Optimal Transport ” aims to make interactive relations via the similarity computation. ”Distribution-Consistent Modality Transfer” is to transform heterogeneous token distributions into a unified latent Gaussian space for subsequent cross-modal interaction. It consists of two fusion processes: LiDAR & Camera Two-Way Interactive Fusion and Language Navigation Prompts Cross-Modal Interactive Fusion.

\tilde{V}=V\oplus{\mathbf{P}^{*}}\cdot B,~~\tilde{B}=B\oplus{({\mathbf{P}^{*}})^{T}}\cdot V

Algorithm 1 Multi-modality Interaction

Fig. 3: Overview of our multi-modal multi-trajectory planning and trajectory refinement. Given the visual, BEV, Text and vehicle ego-vehicle state tokens, we first generate a sequence of continuous trajectories with an 8-waypoint, and further optimize each trajectory by building a driving risk cost map (including detection risk cost map & semantic risk cost map), using real-time 3D detection and BEV map segmentation.

Fig. 4: Visualization comparisons of our method on NAVSIM with different scenes. We conducted a comparison of our method with traditional approach: Goal-Flow and VLA-based approach: WAM-Flow. We use green to represent the ground-truth trajectory.

Traditional End-to-End Manner
Method	Paradigm	Backbone	Input	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
Transfuser [ 17 ] [TPAMI2022]	-	-	3×Cam + LiDAR	97.7	92.8	92.8	100	79.2	84.0
GoalFlow [ 72 ] [CVPR2025]	-	-	3×Cam + LiDAR	98.4	98.3	94.6	100	85.0	90.3
VADv2 [ 34 ] [ICLR2024]	-	-	6×Cam	97.2	89.1	91.6	100	76.0	80.9
DiffusionDrive [ 37 ] [CVPR2025]	Diff.	-	3×Cam + LiDAR	98.2	96.0	94.8	100	82.2	88.1
Artemis [ 47 ] [RAL2025]	Diff.	-	6×Cam	98.3	95.1	94.3	99.8	81.4	87.0

TABLE I: Motion prediction performance on the NAVSIM navtest benchmark with original open-loop metrics: PDMS. Diff., Comf., and Cam denote Diffusion, Comfort and Camera input, respectively. The best and second-best VLA-based results are highlighted in bold and underline.

Method	Input	NC $\uparrow$	DAC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	DDC $\uparrow$	LK $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
VADv2 [ 34 ] [ICLR2024]	Camera	97.3	91.7	77.6	92.7	98.2	66.0	97.4	76.6
TransFuser [ 17 ] [TPAMI2022]	Camera & LiDAR	97.7	92.8	79.2	92.8	98.3	67.6	95.3	77.8
DiffusionDrive [ 37 ] [CVPR2025]	Camera & LiDAR	98.2	96.2	87.6	97.3	98.6	97.0	98.4	84.0
GaussianFusion [ 73 ] [NeurIPS2026]	Camera & LiDAR	98.3	97.3	87.5	97.4	99.0	97.4	98.3	85.0
Ours	Camera & LiDAR	99.0	98.5	88.6	98.3	99.0	97.6	98.7	87.0

TABLE II: Motion prediction performance on the NAVSIM navtest benchmark with extended open-loop metrics: EPDMS. ”Camera” and ”LiDAR” denote Camera and LiDAR inputs, respectively. The best and second-best results are highlighted in bold and underline.

Traditional End-to-End Manner
Method	Input Modality	Core Metrics $\uparrow$				Subdivision Infraction Score $\downarrow$
Method	Input Modality	DS	RC	IS	SR	CP	CV	CL	RL	SS	OR	AB	YEV
TransFuser [ 17 ] [TPAMI2022]	Camera & LiDAR	37.18	68.14	0.51	9.09	0.96	13.24	8.71	0.00	0.96	0.32	2.58	0.32
ThinkTwice [ 24 ] [CVPR2023]	Camera & LiDAR	58.79	74.35	0.77	29.54	0.30	5.76	0.91	0.00	0.91	0.05	0.91	0.30
TCP [ 27 ] [NeurIPS2022]	Camera	56.28	83.57	0.65	25.00	0.26	5.46	5.46	0.00	0.52	0.22	0.78	0.00
NEAT [ 75 ] [ICCV2021]	Camera	30.86	55.35	0.55	6.81	1.08	9.87	5.57	0.20	1.33	0.41	2.01	0.27

TABLE III: Motion prediction performance on Bench2Drive benchmark with core metrics and subdivision infraction scores. DS, RC, IS, SR correspond to the Driving Score, Route Completion, Infraction Score, and Success Rate. CP, CV, CL, RL, SS, OR, AB, YEV correspond to the Collision with a Pedestrian, Collision with another Vehicle, Collision with Layout, Red Light infractions, Stop Sign infractions, Off-Road infractions, Agent Blocked, and failure to Yield to Emergency Vehicles infractions. All comparison results are from VLR-Driver [ 74 ] . The best and second-best VLM/VLA-based results are highlighted in bold and underline.

Traditional Modular End-to-End Paradigm
Method	Ego Status		L2 (m) $\downarrow$				Collision (%) $\downarrow$				Intersection (%) $\downarrow$
Method	BEV	Planner	1s	2s	3s	Avg.	1s	2s	3s	Avg.	1s	2s	3s	Avg.
UniAD [ 30 ] [CVPR2023]	✓	✓	0.20	0.42	0.75	0.46	0.02	0.25	0.84	0.37	0.20	1.33	3.24	1.59
VAD-Base [ 31 ] [ICCV2023]	✓	✓	0.17	0.34	0.60	0.37	0.04	0.27	0.67	0.33	0.21	2.13	5.06	2.47
BEV-Planner [ 33 ] [CVPR2024]	✓	✓	0.16	0.32	0.57	0.35	0.00	0.29	0.73	0.34	0.35	2.62	6.51	3.16
UAD [ 77 ] [TPAMI2025]	✓	✓	0.13	0.28	0.48	0.30	0.00	0.19	0.16	0.27	0.13	1.08	2.89	1.37

TABLE IV: Motion prediction performance on nuScenes benchmark with open-loop metrics. Results are highlighted in bold and underline for the best and second-best performance among VLM/VLA-based methods. All comparison results follow OmniDrive and are from SpaceDrive [ 39 ] .

Method	Modality	validation		test
Method	Modality	mAP $\uparrow$	NDS $\uparrow$	mAP $\uparrow$	NDS $\uparrow$
CenterPoint [ 80 ]	L	59.6	66.8	60.3	67.3
GeoFormer [ 81 ]	L+C	68.2	72.4	69.8	73.7
TransFusion [ 82 ]	L+C	67.5	71.3	68.9	71.6
BEVFusion [ 64 ]	L+C	67.9	71.0	69.2	71.8
BEVFormer [ 25 ]	L+C	71.2	73.2	72.9	74.1

TABLE V: Scene perception performance comparison of 3D object detection on nuScenes test and val set. ”L” and ”C” indicate LiDAR and camera.

Method	mAP	Veh.	Bus	Ped.	Truck	Bicyclist
CenterPoint [ 80 ] [CVPR2021]	22.0	67.6	38.9	46.5	22.1	20.1
VoxelNeXt [ 83 ] [CVPR2023]	30.7	72.7	38.8	63.2	16.9	32.4
HEDNet [ 84 ] [NeurIPS2023]	37.1	78.2	47.7	67.6	21.6	38.7
FSDv2 [ 85 ] [TPAMI2024]	37.6	77.0	47.6	70.5	24.0	45.9
SAFDNet [ 86 ] [CVPR2024]	39.7	78.5	49.4	70.7	23.6	42.7
GeoFormer [ 81 ] [ICCV2025]	41.7	77.4	50.7	73.7	24.6	48.0

TABLE VI: Scene perception performance comparison of 3D object detection on Argoverse 2. All comparison results are all from GeoFormer [ 81 ] .

Fig. 5: Visualization comparisons of both 3D object detection and map segmentation tasks on NuScenes dataset. We conducted a comparison of our method with BEVFusion. In object detection task, we use blue and green to represent the prediction boxes and the ground-truth boxes.

Method	PointPillars [ 21 ]	CenterPoint [ 80 ]	MVP [ 87 ]	BEVFusion [ 64 ]	Ours
Modality	L	L	C+L	C+L	C+L
Drivable	72.0	75.6	76.1	85.5	88.2
Ped.Cross.	43.1	48.4	48.7	60.5	73.4
Walkway	53.1	57.5	57.0	67.6	82.1
Stop Line	29.7	36.5	36.9	52.0	60.9
Carpark	27.7	31.7	33.0	57.0	62.4

TABLE VII: Scene perception comparison of BEV segmentation on the nuScenes val set using the mIoU metric. ”L” and ”C” indicate LiDAR and camera.

Method	Simple-BEV [ 89 ]	BEVFusion [ 64 ]	CMGFA [ 88 ]	Ours
Drivable	72.3	80.8	85.9	87.5
Bicycle	50.1	56.0	60.8	63.8
Vehicle	53.2	63.2	69.3	70.7
Stop sign	42.3	52.2	56.8	55.1
Pedestrain	45.1	51.2	55.2	59.4
Bollard	42.1	48.3	52.1	46.5

TABLE VIII: Scene perception comparison of BEV segmentation on Argoverse 2 dataset. All comparison results are all from CMGFA [ 88 ] .

Fig. 6: Ablation comparisons of trajectory optimization on NAVSIM dataset. We conducted a comparison of our multi-trajectory generation and DFM-based manner along with optimization module.

Planning Loss			Evaluation Metrics
$L_{match}$	$L_{diversity}$	$L_{cons.}$	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
✓			99.0	97.0	97.6	100	85.0	86.9
✓	✓		99.0	97.2	97.8	100	85.4	87.5
✓	✓	✓	99.0	98.5	98.3	100	88.6	92.2

TABLE IX: Ablation study of planning losses

\mathcal{L}_{plan}

on NAVSIM dataset.

VLA Backbone	Evaluation Metrics
VLA Backbone	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
Qwen2.5-3B	98.9	98.4	98.0	99.9	88.5	91.8
LLaVA-7B	99.2	98.3	98.2	100	88.4	92.1
Janus-1.5B	99.0	98.5	98.3	100	88.6	92.2

TABLE X: Ablation study of different VLA foundations on NAVSIM dataset.

Optimization Objectives				Evaluation Metrics
Prior	DRCM	SRCM	Smooth	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
✓	✓	✓	✓	99.0	98.5	98.3	100	88.6	92.2
✗	✓	✓	✓	97.3	96.9	96.2	100	87.8	88.0
✓	✗	✓	✓	96.1	95.7	94.9	99.7	87.2	85.1
✓	✓	✗	✓	96.7	96.0	95.4	98.9	86.4	85.6
✓	✓	✓	✗	98.9	98.5	98.3	97.6	88.0	91.5

TABLE XI: Ablation study of Perception-Oriented Trajectory Refinement on NAVSIM dataset. ”DRCM” and ”SRCM” denote Detection Risk Cost Map & Semantic Risk Cost Map, respectively.

Fig. 8: Visualization of confusion matrices for evaluating our multi-modal interaction on the nuScenes dataset for 3D detection and BEV map segmentation, and on the NAVSIM test set for trajectory planning. We compare the full model with its variants without multi-modal interaction.

Interactive Manner	nuScenes Dataset			NAVSIM Dataset
Interactive Manner	mAP $\uparrow$	NDS $\uparrow$	mIoU $\uparrow$	PDMS $\uparrow$	EPDMS $\uparrow$
BEVFusion	69.2	71.8	62.7	90.3	85.7
LiDAR-only counterpart	70.7	75.9	68.0	91.0	86.4
w/o DCMT	73.6	77.7	71.0	92.0	86.9
w/o prompts interaction	73.9	77.9	71.1	90.1	86.0
Ours	74.0	78.0	71.1	92.2	87.0

TABLE XII: Ablation study of different interaction manners on both NAVSIM and nuScenes dataset. Where ”DCMT” denotes Distribution-Consistent Modality Transfer.

AGOT		DCMT		nuScenes Dataset			NAVSIM Dataset
${\mathcal{L}_{reg}}$	${\mathcal{L}_{ot}}$	${\mathcal{L}_{unf}}$	${\mathcal{L}_{consist}}$	mAP $\uparrow$	NDS $\uparrow$	mIoU $\uparrow$	PDMS $\uparrow$	EPDMS $\uparrow$
✓	✗	✓	✓	72.0	75.4	68.6	91.0	85.8
✗	✓	✓	✓	71.3	76.0	67.5	91.1	86.0
✓	✓	✓	✗	73.0	77.5	70.7	91.8	86.7
✓	✓	✗	✓	73.2	77.7	70.9	91.7	86.4
✓	✓	✓	✓	74.0	78.0	71.1	92.2	87.0

TABLE XIII: Ablation study of our multi-modality interaction on NAVSIM and nuScenes. ”AGOT” and ”DCMT” denote Affinity-Guided Optimal Transport and Distribution-Consistent Modality Transfer.

Explore similar work

VECTOR-Drive: Tightly Coupled Vision-Language and Trajectory Expert Routing for End-to-End Autonomous Driving
May 9, 2026Rui Zhao, Jianlin Yu, Zhenhai Gao +2VLMs for Autonomous Driving End-to-End Autonomous Driving
College of Automotive Engineering, Jilin University · 2The National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University · 3ReeFocus AI Technology
UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
Apr 2, 2026Yongkang Li, Lijun Zhou, Sixu Yan +11Vision-Language-Action Models End-to-End Autonomous Driving
Huazhong University of Science and Technology · Xiaomi EV · SKL-IOTSC, University of Macau
MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving
May 12, 2026Yuzhou Huang, Benjin Zhu, Hengtong Lu +6VLMs for Autonomous Driving End-to-End Autonomous Driving
CUHK MMLab · Li Auto · Tsinghua University

May 9, 2026J/K move · Enter open · S save

VECTOR-Drive: Tightly Coupled Vision-Language and Trajectory Expert Routing for End-to-End Autonomous Driving

Rui Zhao, Jianlin Yu, Zhenhai Gao, Jiaqiao Liu+1

College of Automotive Engineering, Jilin University · 2The National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University · 3ReeFocus AI Technology

VLMs for Autonomous Driving End-to-End Autonomous Driving Vision-Language-Action Models Autonomous Driving Planning Mixture-of-Experts Models

Open paper AskPDF arXiv

End-to-end autonomous driving requires models to understand traffic scenes, infer driving intent, and generate executable motion plans. Recent vision-language-action (VLA) models inherit semantic priors from large-scale vision-language pretraining, yet still face a coupling trade-off: fully shared backbones preserve multimodal interaction but may entangle language reasoning and trajectory prediction, whereas decou pled reasoning-action pipelines reduce task conflict but weaken semantic-motion coupling. We propose VECTOR-DRIVE, a tightly coupled VLA framework built on Qwen2.5-VL-3B. VECTOR-DRIVE keeps all tokens coupled through shared self attention and routes feed-forward computation according to token semantics. Vision and language tokens are processed by a Vision-Language Expert to preserve semantic priors, while target-point, ego-state, and noisy action tokens are routed to a Trajectory Expert for motion-specific computation. On the action-token pathway, a flow-matching planner refines noisy action tokens into future waypoints and speed profiles. This design couples semantic reasoning and motion planning within a single multimodal Transformer while separating task-specific FFN computation. On Bench2Drive, VECTOR-DRIVE achieves 88.91 Driving Score and outperforms representative end-to end and VLA-based baselines. Qualitative results and ablations further validate the benefits of shared attention, semantic-aware expert routing, progressive training, and flow-based action de coding.

Abstract

Explore similar work

VECTOR-Drive: Tightly Coupled Vision-Language and Trajectory Expert Routing for End-to-End Autonomous Driving

UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving

MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving