Vision-language-action (VLA) models have advanced end-to-end autonomous driving by leveraging foundation models for semantic reasoning and long-tail generalization. However, their planning performance remains limited in complex driving environments because image-only representations inadequately capture planning-relevant road geometry and topology. In this paper, we propose Geo-VLA, a plug-and-play framework that enhances VLA models by learning geometry-aware visual representations. During training, Geo-VLA internalizes geometric map semantics to strengthen road-structure representations, while requiring no HD maps or additional lane information during inference. To support this approach, we introduce Geo-QA, a geometry-focused question-answering dataset that injects road geometry into vision-language representations through contrastive learning and instruction tuning. Experiments on NAVSIM v1 demonstrate that Geo-VLA consistently improves VLA planners with distinct action-generation architectures, achieving 92.1 PDMS and establishing a new state-of-the-art among single-camera VLA planners.
Figures & tables
Figure 1: Representative VLA planning failures in turning scenes. Each row shows the front-view observation on the left and the corresponding bird’s-eye view on the right. At intersections and curves, the predicted trajectory deviates from the expert trajectory and the intended road geometry.
Figure 2: Overview of Geo-VLA for end-to-end VLA planning. During geometry-aware pretraining, a front-view image and its paired offline map annotation are converted into a Geo-QA pair to adapt the vision-language backbone from Φθ to Φ∗ . The corresponding hidden states change from Ft to the geometry-aware representation Ftgeo . During planning fine-tuning, Φ∗ is fixed and the original action decoder is optimized with expert-trajectory supervision under its original training objective. Here, Φ∗ denotes the adapted vision-language backbone after pretraining, excluding the auxiliary projection heads used only for pretraining. At inference time t , the inputs remain the front-view image It , ego history Ht , ego state st , and navigation command ct . The decoder predicts Y^tgeo=[w^t,…,w^t+n] , where n is the planning horizon. Offline map annotations and Geo-QA pairs are used only during training.
Figure 3: Geo-QA construction and the two-stage training of Geo-VLA. The left part shows how the Geo-QA generator uses a front-view image Ii , its local offline map annotation Mi , and a predefined category ri to obtain a question Qi and an answer Ai with GPT-5.4. The label A/T indicates that the same answer Ai also serves as the textual description Ti of map semantics for contrastive learning. The right part presents the stage-specific learning objectives and model architecture. Here, i indexes a Geo-QA sample. Geometry-aware pretraining jointly performs contrastive alignment and question-answer instruction tuning to optimize the trainable parts of the vision encoder and LLM backbone. Planning fine-tuning then optimizes the action decoder through the planner’s original imitation-learning objective. The adapted backbone produces geometry-aware hidden states Ftgeo , from which the decoder predicts Y^tgeo . Flame icons identify the components optimized at each stage.
Method
NC ↑
DAC ↑
TTC ↑
C ↑
EP ↑
PDMS ↑
ReCogDrive † ( 13 )
98.20
97.50
94.80
100.00
87.50
90.8
DriveFine ( 3 )
98.80
98.60
96.20
100.00
86.90
91.8
LaST-VLA ( 19 )
98.70
97.90
95.60
100.00
86.80
91.3
DynVLA † ( 21 )
98.00
97.20
94.20
100.00
85.20
91.0
ReCogDrive + Geo-VLA
98.40
98.00
95.00
100.00
88.00
91.2
DynVLA + Geo-VLA
98.50
98.20
95.30
100.00
89.00
92.1
Table 1: Comparison with single-camera VLA planners on NAVSIM v1 navtest under the standard single-trajectory protocol. Best-of- N and candidate-selection methods are excluded. NC: No at-fault Collisions; DAC: Drivable Area Compliance; TTC: Time-to-Collision; C: Comfort; EP: Ego Progress; PDMS: Predictive Driver Model Score. † denotes our reproduction. Bold and underline denote the best and second-best results, respectively.
VLA model
Geometry variant
Inference Overhead
NC ↑
DAC ↑
TTC ↑
C ↑
EP ↑
PDMS ↑
ReCogDrive
Baseline
Same as baseline
98.20
97.50
94.80
100.00
87.50
90.8
HD map encoder
High
98.50
99.20
95.20
100.00
88.80
91.9
Lane detection
Moderate
98.30
98.10
94.50
100.00
88.20
91.2
Geo-QA
Same as baseline
98.40
98.00
95.00
100.00
88.00
91.2
DynVLA
Baseline
Same as baseline
98.00
97.20
94.20
100.00
85.20
91.0
HD map encoder
High
98.80
98.50
95.20
100.00
89.20
92.4
Table 2: Comparison of different map encoding approaches on NAVSIM v1 navtest. Inference Overhead is relative to the baseline: Same as baseline adds no module, Moderate adds online lane detection, and High requires online map localization, retrieval, and encoding. Metric abbreviations follow Table 1 .
Figure 4: Qualitative planning results in representative turning scenes. In the left panel, each row shows the front-view image, ReCogDrive prediction, and Geo-VLA prediction from left to right. In the right panel, each row shows the front-view image, lane-detection variant, and Geo-QA variant from left to right. Red and green denote predicted and expert trajectories, respectively.
Model
QA type
PDMS ↑
DynVLA
None
91.0
Dynamic
90.7
Static
92.1
Table 3: Effect of static and dynamic QA supervision.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: A representative Geo-QA record shown in reader-facing form. The sample pairs a sampled front-view image with a geometry-focused question and a map-grounded answer.
Vision-language-action~(VLA) models have emerged as a promising paradigm for autonomous driving. However, existing VLA models still suffer from a fundamental mismatch: driving actions require precise 3D geometric cues, while visual-language understanding and reasoning are largely conducted in a 2D semantic space. In this paper, we propose GeoCoTDrive, an explicit geometric chain-of-thought framework that grounds geometry in a planning-oriented manner. GeoCoTDrive follows a think with 2D first, drive with dedicated 3D priors paradigm. It first grounds 2D regions corresponding to decision-critical cues, and then retrieves localized 3D priors by sampling features from a geometric foundation model within the grounded regions. These localized geometric features are interleaved into the autoregressive context to support the trajectory generation. To supervise this process, we introduce planning-relevant grounding, a new region-level grounding task that focuses on local spatial cues directly affecting ego planning decisions, and construct the PlanningGrounding dataset to endow VLAs with planning-oriented grounding capability. Experiments across multiple end-to-end autonomous driving benchmarks show that GeoCoTDrive consistently improves safety-critical planning performance, demonstrating the effectiveness of the explicit geometric chain-of-thought process for VLA-based planning.
Xingtai Gui, Yucheng Zhou, Dongqian Guo +3
SKL-IOTSC, Department of AI, University of Macau · Afari Intelligent Drive
Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.
Yuguang Yang, Canyu Chen, Zhewen Tan +10
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5
Vision-language-action (VLA) models can describe scenes and reason about them in language, yet still struggle to ground their actions in the dense 3D world around them. Existing approaches either inject features from a frozen 3D foundation model without an objective that ensures the policy uses them, or constrain geometry with sparse box and map losses that provide no dense spatial signal. We introduce VLGA, the first vision-language-action model supervised to reconstruct the dense 3D world it drives through. VLGA introduces geometry as a fourth modality alongside vision, language, and action through a dedicated expert supervised by a per-pixel pointmap regression loss against LiDAR. Extensive experiments conducted on challenging nuScenes and Bench2Drive datasets for open-loop and closed-loop evaluations, respectively, show the superiority of VLGA over counterpart VLA methods. In particular, on open-loop nuScenes, VLGA sets a new state of the art among VLA methods without ego status, with the lowest L2 (0.50,m average) and 3-second collision rate (0.18%). On closed-loop Bench2Drive, VLGA attains the state-of-the-art driving score of 79.08, +0.71 over the strongest prior VLA, at comparable efficiency and comfort.