cs.LGSep 10, 2026

Grounding Vision-Language Models in Driving Semantics: A Multi-Dataset Predicate Framework for Explainable Reasoning

Authors: Mohamed Chouai, Fazli Faruk Okumus, Stefan Kugele

Organizations: AImotion Bavaria, Technische Hochschule Ingolstadt, 85049 Ingolstadt, Germany.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-language models are increasingly used for driving-scene understanding, yet the semantic relations expressed in their outputs are often difficult to verify against the underlying traffic situation. This paper introduces a deterministic multi-dataset predicate framework that derives driving-scene semantics from measurable geometric, kinematic, temporal, map, and traffic-control evidence. Dataset-specific interfaces are used only to recover the required scene information, while predicate definitions remain unchanged across nuPlan and nuScenes and are materialised in a common Predicate Knowledge Graph. Quantitative semantic validation against manually annotated predicate relations on 200 scenarios from each dataset yields macro F1 scores of 0.94 on nuPlan and 0.93 on nuScenes, with an average cross-dataset difference of 0.02 across the shared predicates. The Predicate KG is further evaluated using a frozen LLaVA-OneVision-7B model on the nine NuPlanQA subtasks. Predicate grounding achieves the highest accuracy among the evaluated visual-input conditions in seven of nine NuPlanQA subtasks, including Traffic Light (53.2% to 71.5%), Situation Assessment (76.2% to 86.1%), and Action Recommendation (82.9% to 89.0%). Weather/Lighting remains essentially unchanged (89.4% vs. 88.8%), consistent with the absence of corresponding predicates, while Predicate KG only input outperforms metadata-only input in eight of nine subtasks. The results show that deterministic predicates provide a consistent and traceable semantic representation and, under oracle grounding, can reduce visual dependence for reasoning tasks covered by the predicate vocabulary.

Figures & tables

Fig. 1 : Overview of the proposed framework. Measurable driving-scene evidence is processed through deterministic predicate rules and materialised in a Predicate KG. The example illustrates the correspondence between a physical traffic scene and its semantic predicate representation.
PredicateOperational summary
follows , queuesBehindPersistent longitudinal relation between participants on a compatible travel path; queuesBehind additionally captures queued traffic.
changesLaneTemporal transition of a participant between lane memberships during a lane-change manoeuvre.
mergesInFrontOf , mergesBehindMerge into a common travel path, distinguished by the resulting longitudinal ordering of the participants.
overtakesTemporal manoeuvre in which the relative ordering changes from behind to ahead.
yieldsToInteraction in which a participant gives precedence to another participant during a conflicting movement.
TABLE I : Operational summary of the evaluated higher-level predicates.
 nuPlannuScenes 
PredicateN\boldsymbol{N}F1\boldsymbol{F_{1}}N\boldsymbol{N}F1\boldsymbol{F_{1}}ΔF1\boldsymbol{\Delta F_{1}}
follows4700.984300.970.01
queuesBehind2600.992200.990.00
changesLane1650.921500.950.03
mergesInFrontOf1050.96950.930.03
mergesBehind850.94750.930.01
TABLE II : Semantic validation of higher-level derived predicates. NN denotes the number of reviewed candidates before exclusion of ambiguous cases.
 Road Env. PerceptionSpatial Relations Recog.Ego-centric Reasoning
MethodTraffic LightWeather/LightingRoad TypeSur. Obj.Traffic FlowKey Obj.Ego Ctrl.Situ. Asse.Act. Rec.
Visual input
LLaVA-OV-7B53.289.496.477.977.673.275.476.282.9
+ GT Metadata62.889.096.583.081.077.563.481.084.0
+ Predicate KG71.588.896.882.785.882.383.986.189.0
No visual input
TABLE III : Effect of structured scene grounding on the nine NuPlanQA-Eval subtasks. Bold indicates the best result among conditions with visual input; underlining indicates the best result without visual input. All values are accuracy (%).
Fig. 2 : Representative predicate relations at three selected time steps for three predicate families shown as illustrative examples: spatial, interaction, and traffic-control. Colours denote predicate semantics, and the black arrow indicates the ego vehicle’s travel direction.
More in this topic

Vision-Language Grounding

Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.

See all 420 papers

Explore similar work

CardsList
Sep 28, 2026

Beyond Retrieval Relevance: Scene-Grounded Risk Entailment for Vision-Language Driving

Retrieval-augmented generation (RAG) gives vision--language driving systems access to external safety knowledge, yet a retrieved risk rule may be relevant without applying to the current scene. A vision--language model (VLM) receiving such knowledge must ground objects, bind entities across time, and verify relations before deciding how to act, leaving the support for risk conclusions implicit. We address this relevance--applicability gap with a Driving-Risk Knowledge Graph (DRKG) and Semantic Web Rule Language (SWRL) reasoning stage before VLM decision-making. Structured perception instantiates scene facts, from which SWRL rules derive events and directed risk relations when their antecedents are jointly satisfied. Recognized events, bound risk relations, and semantic descriptions of activated rules form compact evidence that conditions the VLM and diffusion planner. In matched comparisons on nuReasoning, our method improved the nuReasoning planning score (NPS) by 1.30 points and the non-at-fault collision score (NC) by 2.76 points over the relevance retrieval-based baseline. These gains indicate that scene-applicable risk evidence improves safety-weighted planning relative to semantically retrieved risk knowledge.

Jiaxin Liu, Ruilin Yu, Liang Peng +8

School of Vehicle and Mobility, Tsinghua University · Xiaomi EV · Jilin University

VLMs for Autonomous DrivingAutonomous DrivingVLM ReasoningAutonomous Driving Planning
AskPDF
Jun 5, 2026

VeriDrive: Verifiable Counterfactual Supervision for Cost-Efficient Vision-Language Planning

Vision-language driving models increasingly use reasoning supervision to bridge perception, prediction, and planning, but existing driving rationales are often free-form and expensive to generate with frontier models. We present VeriDrive, a framework for constructing planning-oriented, verifiable counterfactual supervision. VeriDrive converts driving reasoning into a structured Perception-Evaluation-Revision chain that grounds key objects in future motion, evaluates alternative ego trajectories with rule-checkable evidence, revises risky intent toward expert behavior, and produces final planning targets. To scale data construction, VeriDrive combines local generation with validator-guided selective correction, escalating only invalid or difficult samples. We build the VeriDrive dataset on nuScenes and train under the Omni-Q protocol. Controlled open-loop experiments show that VeriDrive improves L2, Collision, and Intersection over OmniDrive while reducing logged token usage, generation time, and actual paid LLM/VLM cost. These results show that auditable intermediate fields and structured revision targets can improve vision-language planning supervision under realistic annotation budgets. Code, prompts, and validator scripts are coming soon and will be released after the review process.

Zikai Zhang, Hubert P. H. Shum, Toby P. Breckon

Department of Computer Science Durham University Stockton Road, Durham, DH1 3LE, UK

Autonomous Driving DatasetsVLMs for Autonomous DrivingAutonomous DrivingVLM ReasoningAutonomous Driving Planning
AskPDF
Oct 6, 2025

More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models

Vision-Language Model (VLM) driving agents promise explainable end-to-end autonomy by first producing natural-language reasoning and then predicting trajectory planning. However, whether planning is causally driven by this reasoning remains a critical but unverified assumption. To investigate this, we build DriveMind, a large-scale driving Visual Question Answering corpus with plan-aligned Chain-of-Thought (CoT), automatically generated from nuPlan. Our data generation process converts sensors and annotations into structured inputs and, crucially, separates priors from to-be-reasoned signals, enabling clean information ablations. Using DriveMind, we train representative VLM agents with Supervised Fine-Tuning and Group Relative Policy Optimization and evaluate them with nuPlan's metrics. Our results, unfortunately, indicate a consistent causal disconnect in reasoning-planning: removing ego/navigation priors causes large drops in planning scores, whereas removing CoT produces only minor changes. Attention analysis further shows that planning primarily focuses on priors rather than the CoT. Based on this evidence, we propose the Reasoning-Planning Decoupling Hypothesis, positing that the training-yielded reasoning is an ancillary byproduct rather than a causal mediator. To enable efficient diagnosis, we introduce a novel, training-free probe that measures an agent's reliance on priors by evaluating its planning robustness against minor input perturbations. In summary, we provide the community with a new dataset and a diagnostic tool to evaluate the causal fidelity of future models.

Xurui Song, Shuo Huai, JingJing Jiang +2

S-Lab, Nanyang Technological University, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore

VLMs for Autonomous DrivingReasoning for VLA ModelsAutonomous Driving PlanningFaithfulness in VLMs
AskPDF