Geo-VLA: Geometry-Aware Vision-Language-Action Planning via Internalization of Map Semantics

Ran Chen; Jiaxing Ren; Zhikun Zhang; Yunhao Hou; Junbao Zhuo; Bochao Zou

cs.ROAug 18, 2026

Geo-VLA: Geometry-Aware Vision-Language-Action Planning via Internalization of Map Semantics

Authors: Ran Chen, Jiaxing Ren, Zhikun Zhang, Yunhao Hou, Junbao Zhuo, Bochao Zou

Organizations: University of Science and Technology Beijing Beijing, China

Abstract

Vision-language-action (VLA) models have advanced end-to-end autonomous driving by leveraging foundation models for semantic reasoning and long-tail generalization. However, their planning performance remains limited in complex driving environments because image-only representations inadequately capture planning-relevant road geometry and topology. In this paper, we propose Geo-VLA, a plug-and-play framework that enhances VLA models by learning geometry-aware visual representations. During training, Geo-VLA internalizes geometric map semantics to strengthen road-structure representations, while requiring no HD maps or additional lane information during inference. To support this approach, we introduce Geo-QA, a geometry-focused question-answering dataset that injects road geometry into vision-language representations through contrastive learning and instruction tuning. Experiments on NAVSIM v1 demonstrate that Geo-VLA consistently improves VLA planners with distinct action-generation architectures, achieving 92.1 PDMS and establishing a new state-of-the-art among single-camera VLA planners.

Figures & tables

Figure 1: Representative VLA planning failures in turning scenes. Each row shows the front-view observation on the left and the corresponding bird’s-eye view on the right. At intersections and curves, the predicted trajectory deviates from the expert trajectory and the intended road geometry.

Figure 2: Overview of Geo-VLA for end-to-end VLA planning. During geometry-aware pretraining, a front-view image and its paired offline map annotation are converted into a Geo-QA pair to adapt the vision-language backbone from $\Phi_{\theta}$ to $\Phi^{*}$ . The corresponding hidden states change from $F_{t}$ to the geometry-aware representation $F_{t}^{\mathrm{geo}}$ . During planning fine-tuning, $\Phi^{*}$ is fixed and the original action decoder is optimized with expert-trajectory supervision under its original training objective. Here, $\Phi^{*}$ denotes the adapted vision-language backbone after pretraining, excluding the auxiliary projection heads used only for pretraining. At inference time $t$ , the inputs remain the front-view image $I_{t}$ , ego history $H_{t}$ , ego state $s_{t}$ , and navigation command $c_{t}$ . The decoder predicts $\hat{\mathbf{Y}}_{t}^{\mathrm{geo}}=[\hat{\mathbf{w}}_{t},\ldots,\hat{\mathbf{w}}_{t+n}]$ , where $n$ is the planning horizon. Offline map annotations and Geo-QA pairs are used only during training.

Figure 3: Geo-QA construction and the two-stage training of Geo-VLA. The left part shows how the Geo-QA generator uses a front-view image $I_{i}$ , its local offline map annotation $\mathcal{M}_{i}$ , and a predefined category $r_{i}$ to obtain a question $Q_{i}$ and an answer $A_{i}$ with GPT-5.4. The label A/T indicates that the same answer $A_{i}$ also serves as the textual description $T_{i}$ of map semantics for contrastive learning. The right part presents the stage-specific learning objectives and model architecture. Here, $i$ indexes a Geo-QA sample. Geometry-aware pretraining jointly performs contrastive alignment and question-answer instruction tuning to optimize the trainable parts of the vision encoder and LLM backbone. Planning fine-tuning then optimizes the action decoder through the planner’s original imitation-learning objective. The adapted backbone produces geometry-aware hidden states $F_{t}^{\mathrm{geo}}$ , from which the decoder predicts $\hat{\mathbf{Y}}_{t}^{\mathrm{geo}}$ . Flame icons identify the components optimized at each stage.

Method	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	C $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
ReCogDrive † ( 13 )	98.20	97.50	94.80	100.00	87.50	90.8
DriveFine ( 3 )	98.80	98.60	96.20	100.00	86.90	91.8
LaST-VLA ( 19 )	98.70	97.90	95.60	100.00	86.80	91.3
DynVLA † ( 21 )	98.00	97.20	94.20	100.00	85.20	91.0
ReCogDrive + Geo-VLA	98.40	98.00	95.00	100.00	88.00	91.2
DynVLA + Geo-VLA	98.50	98.20	95.30	100.00	89.00	92.1

Table 1: Comparison with single-camera VLA planners on NAVSIM v1 navtest under the standard single-trajectory protocol. Best-of-

N

and candidate-selection methods are excluded. NC: No at-fault Collisions; DAC: Drivable Area Compliance; TTC: Time-to-Collision; C: Comfort; EP: Ego Progress; PDMS: Predictive Driver Model Score.

\dagger

denotes our reproduction. Bold and underline denote the best and second-best results, respectively.

VLA model	Geometry variant	Inference Overhead	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	C $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
ReCogDrive	Baseline	Same as baseline	98.20	97.50	94.80	100.00	87.50	90.8
	HD map encoder	High	98.50	99.20	95.20	100.00	88.80	91.9
	Lane detection	Moderate	98.30	98.10	94.50	100.00	88.20	91.2
	Geo-QA	Same as baseline	98.40	98.00	95.00	100.00	88.00	91.2
DynVLA	Baseline	Same as baseline	98.00	97.20	94.20	100.00	85.20	91.0
	HD map encoder	High	98.80	98.50	95.20	100.00	89.20	92.4

Table 2: Comparison of different map encoding approaches on NAVSIM v1 navtest. Inference Overhead is relative to the baseline: Same as baseline adds no module, Moderate adds online lane detection, and High requires online map localization, retrieval, and encoding. Metric abbreviations follow Table 1 .

Figure 4: Qualitative planning results in representative turning scenes. In the left panel, each row shows the front-view image, ReCogDrive prediction, and Geo-VLA prediction from left to right. In the right panel, each row shows the front-view image, lane-detection variant, and Geo-QA variant from left to right. Red and green denote predicted and expert trajectories, respectively.

Model	QA type	PDMS $\uparrow$
DynVLA	None	91.0
	Dynamic	90.7
	Static	92.1

Table 3: Effect of static and dynamic QA supervision.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: A representative Geo-QA record shown in reader-facing form. The sample pairs a sampled front-view image with a geometry-focused question and a map-grounded answer.

Explore similar work

Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving
Oct 7, 2026Xingtai Gui, Yucheng Zhou, Dongqian Guo +3Autonomous Driving Planning Reasoning for VLA Models
SKL-IOTSC, Department of AI, University of Macau · Afari Intelligent Drive
Teaching Vision-Language-Action Models What to See and Where to Look
Jul 2, 2026Yuguang Yang, Canyu Chen, Zhewen Tan +10End-to-End Autonomous Driving VLM Distillation
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5
VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving
Jun 10, 2026Jin Yao, Dhruva Dixith Kurra, Tom Lampo +3VLMs for Autonomous Driving End-to-End Autonomous Driving
Uber AV Labs · University of Virginia

Oct 7, 2026J/K move · Enter open · S save

Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving

Xingtai Gui, Yucheng Zhou, Dongqian Guo, Jiahao Gong+2

SKL-IOTSC, Department of AI, University of Macau · Afari Intelligent Drive

Autonomous Driving Planning Reasoning for VLA Models

Open paper AskPDF arXiv

Vision-language-action~(VLA) models have emerged as a promising paradigm for autonomous driving. However, existing VLA models still suffer from a fundamental mismatch: driving actions require precise 3D geometric cues, while visual-language understanding and reasoning are largely conducted in a 2D semantic space. In this paper, we propose GeoCoTDrive, an explicit geometric chain-of-thought framework that grounds geometry in a planning-oriented manner. GeoCoTDrive follows a think with 2D first, drive with dedicated 3D priors paradigm. It first grounds 2D regions corresponding to decision-critical cues, and then retrieves localized 3D priors by sampling features from a geometric foundation model within the grounded regions. These localized geometric features are interleaved into the autoregressive context to support the trajectory generation. To supervise this process, we introduce planning-relevant grounding, a new region-level grounding task that focuses on local spatial cues directly affecting ego planning decisions, and construct the PlanningGrounding dataset to endow VLAs with planning-oriented grounding capability. Experiments across multiple end-to-end autonomous driving benchmarks show that GeoCoTDrive consistently improves safety-critical planning performance, demonstrating the effectiveness of the explicit geometric chain-of-thought process for VLA-based planning.

Abstract

Explore similar work

Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving

Teaching Vision-Language-Action Models What to See and Where to Look

VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving