Towards Viewpoint-Robust End-to-End Autonomous Driving with 3D Foundation Model Priors

Hiroki Hashimoto; Hiromichi Goto; Hiroyuki Sugai; Hiroshi Kera; Kazuhiko Kawamoto

cs.CVApr 1, 2026

Towards Viewpoint-Robust End-to-End Autonomous Driving with 3D Foundation Model Priors

Authors: Hiroki Hashimoto, Hiromichi Goto, Hiroyuki Sugai, Hiroshi Kera, Kazuhiko Kawamoto

Organizations: Chiba University · National Institute of Informatics

Abstract

Robust trajectory planning under camera viewpoint changes is important for scalable end-to-end autonomous driving. However, existing models often depend heavily on the camera viewpoints seen during training. We investigate an augmentation-free approach that leverages geometric priors from a 3D foundation model. The method injects per-pixel 3D positions derived from depth estimates as positional embeddings and fuses intermediate geometric features through cross-attention. Experiments on the VR-Drive camera viewpoint perturbation benchmark show reduced performance degradation under most perturbation conditions, with clear improvements under pitch and height perturbations. Gains under longitudinal translation are smaller, suggesting that more viewpoint-agnostic integration is needed for robustness to camera viewpoint changes.

Figures & tables

Figure 1: Overview of the proposed method. We extract geometric features and depth estimates from a 3D foundation model and integrate them into an end-to-end autonomous driving model to improve robustness against camera viewpoint changes.

Figure 2: Architecture of the proposed method. (a) 3D Spatial Encoder: computes depth-derived 3D positions from DA3 and camera parameters, and injects them as positional embeddings into image features. (b) Geometric Feature Fusion: fuses DA3 intermediate features into image features via cross-attention.

Figure 3: Camera images under different viewpoint perturbation conditions. Each column shows the Original, Pitch $-10^{\circ}$ , and Depth $+1.0$ m conditions for the front and front-left cameras.

Method	Type	Original		Pitch $+5^{\circ}$		Pitch $-10^{\circ}$		Height $+1.0$ m		Height $-0.7$ m		Depth $+1.0$ m
Method	Type	L2 [m] $\downarrow$	Col. [%] $\downarrow$	L2 [m] $\downarrow$	Col. [%] $\downarrow$	L2 [m] $\downarrow$	Col. [%] $\downarrow$	L2 [m] $\downarrow$	Col. [%] $\downarrow$	L2 [m] $\downarrow$	Col. [%] $\downarrow$	L2 [m] $\downarrow$	Col. [%] $\downarrow$
AD-MLP $*$	-	0.29	0.19	0.29	0.19	0.29	0.19	0.29	0.19	0.29	0.19	0.29	0.19
BEV-Planner $*$	-	0.55	0.22	0.59	0.37	0.54	0.76	0.57	0.29	0.58	0.64	0.58	0.28
VAD	P	0.72	0.22	0.68	0.28	1.02	0.88	0.73	0.47	0.74	0.22	0.71	0.26
SparseDrive	P	0.61	0.08	0.66	0.15	0.96	0.23	0.87	0.54	1.01	0.30	1.27	0.31
DiffusionDrive	P	0.57	0.08	0.67	0.11	0.96	0.24	1.46	0.81	1.21	0.20	1.57	0.41

Table 1: Comparison of trajectory planning performance under each viewpoint perturbation condition in the VR-Drive evaluation framework.

*

denotes methods that use only ego-status without camera images as input. The Type column indicates P for Perception-based and S for Self-Supervised.

Figure 4: BEV trajectory comparison under viewpoint perturbations. Rows correspond to Height

+1.0

m and Depth

+1.0

m conditions. Under Height perturbation, the proposed method remains close to the ground truth, whereas World4Drive shows a larger deviation. Under Depth perturbation, both methods fail.

Depth	GFF	Original		Pitch $+5^{\circ}$		Pitch $-10^{\circ}$		Height $+1.0$ m		Height $-0.7$ m		Depth $+1.0$ m
Depth	GFF	L2 [m] $\downarrow$	Col. [%] $\downarrow$	L2 [m] $\downarrow$	Col. [%] $\downarrow$	L2 [m] $\downarrow$	Col. [%] $\downarrow$	L2 [m] $\downarrow$	Col. [%] $\downarrow$	L2 [m] $\downarrow$	Col. [%] $\downarrow$	L2 [m] $\downarrow$	Col. [%] $\downarrow$
Metric3D		0.48	0.26	0.83	0.48	0.64	0.36	1.06	1.29	5.01	5.44	7.10	6.40
Metric3D	✓	0.58	0.35	0.65	0.38	0.73	0.33	0.83	0.46	0.78	0.34	4.94	5.92
DA3		0.52	0.28	0.70	0.37	1.06	0.71	4.58	4.54	5.27	5.80	7.91	6.73
DA3	✓	0.49	0.32	0.54	0.30	0.69	0.21	0.63	0.42	0.90	0.46	6.61	5.21

Table 2: Ablation of module contributions. The Depth column indicates the depth estimator used in the 3D Spatial Encoder, and the GFF column indicates whether Geometric Feature Fusion is enabled.

Replaced cameras	L2 [m] $\downarrow$	Col. [%] $\downarrow$
None (Depth $+1.0$ m)	6.61	5.21
Front/rear only	5.62	4.81
Four side cameras	1.30	0.30
All cameras	0.51	0.29

Table 3: Counterfactual diagnosis under Depth

+1.0

m. Camera extrinsic parameters used for 3D positional embeddings are replaced with their training-time values at test time.

End-to-End Autonomous Driving

End-to-end autonomous driving learns to turn sensor observations and driving context into vehicle actions or trajectories.

See all 227 papers

VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving
Feb 24, 2026Jie Wang, Guang Li, Zhijian Huang +4Multi-View Feature Fusion VLMs for Autonomous Driving
College of Intelligence and Computing, Tianjin University · Xiaomi EV
Drive3R: Block-wise Streaming Geometric Foundation Models for Autonomous Driving
Oct 10, 2026Tianyi Yang, Haojun Yang, Jipeng Xiao +43D Foundation Models Streaming 3D Reconstruction
AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · Voyager Research, Didi Chuxing · The Chinese University of Hong Kong, Shenzhen
Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving
Aug 31, 2026Xin Zhou, Zongchuang Zhao, Zhibo Yang +13VLMs for Autonomous Driving Autonomous Driving Perception
Qwen Team, Huazhong University of Science and Technology

Feb 24, 2026J/K move · Enter open · S save

VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving

Jie Wang, Guang Li, Zhijian Huang, Chenxu Dang+3

College of Intelligence and Computing, Tianjin University · Xiaomi EV

Multi-View Feature Fusion VLMs for Autonomous Driving 3D VLMs Feature Fusion

Open paper AskPDF arXiv

The significance of cross-view 3D geometric modeling capabilities for autonomous driving is self-evident, yet existing Vision-Language Models (VLMs) inherently lack this capability, resulting in their mediocre performance. While some promising approaches attempt to mitigate this by constructing Q&A data for auxiliary training, they still fail to fundamentally equip VLMs with the ability to comprehensively handle diverse evaluation protocols. We thus chart a new course, advocating for the infusion of VLMs with the cross-view geometric grounding of mature 3D foundation models, closing this critical capability gap in autonomous driving. In this spirit, we propose a novel architecture, VGGDrive, which empowers Vision-language models with cross-view Geometric Grounding for autonomous Driving. Concretely, to bridge the cross-view 3D geometric features from the frozen visual 3D model with the VLM's 2D visual features, we introduce a plug-and-play Cross-View 3D Geometric Enabler (CVGE). The CVGE decouples the base VLM architecture and effectively empowers the VLM with 3D features through a hierarchical adaptive injection mechanism. Extensive experiments show that VGGDrive enhances base VLM performance across five autonomous driving benchmarks, including tasks like cross-view risk perception, motion prediction, and trajectory planning. It's our belief that mature 3D foundation models can empower autonomous driving tasks through effective integration, and we hope our initial exploration demonstrates the potential of this paradigm to the autonomous driving community.

Abstract

End-to-End Autonomous Driving

Explore similar work

VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving

Drive3R: Block-wise Streaming Geometric Foundation Models for Autonomous Driving

Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving