Autonomous driving requires reliable reasoning over fine-grained 3D scene facts. Fine-grained question answering over multi-modal driving observations provides a natural way to evaluate this capability, yet existing perception pipelines and driving-oriented large language model (LLM) methods still suffer from unreliable scene facts, hallucinations, opaque reasoning, and heavy reliance on task-specific training. We present KLDrive, the first knowledge-graph-augmented LLM reasoning framework for fine-grained question answering in autonomous driving. KLDrive addresses this problem through designing two tightly coupled components: an energy-based scene fact construction module that consolidates multi-source evidence into a reliable scene knowledge graph, and an LLM agent that performs fact-grounded reasoning over a constrained action space under explicit structural constraints. By combining structured prompting with few-shot in-context exemplars, the framework adapts to diverse reasoning tasks without heavy task-specific fine-tuning. Experiments on two large-scale autonomous-driving QA benchmarks show that KLDrive outperforms prior state-of-the-art methods, achieving the best overall accuracy of 65.04% on NuScenes-QA and the best SPICE score of 42.45 on GVQA. On counting, the most challenging factual reasoning task, it improves over the strongest baseline by 46.01 percentage points, demonstrating substantially reduced hallucinations and the benefit of coupling reliable scene fact construction with explicit reasoning.
Figures & tables
Figure 1. KLDrive: fact-based scene question answering via a 3D scene KG and a tool-constrained LLM agent.
Methods
Perception ability
Reasoning ability
No task-specific training
Explainable reasoning
Perception-based Methods
RayDN ( Liu et al., 2024a )
✓
✗
✗
✗
FocalFormer3D ( Chen et al., 2023 )
✓
✗
✗
✗
IS-Fusion ( Yin et al., 2024 )
✓
✗
✗
✗
Driving-oriented LLM frameworks
DriveLM ( Sima et al., 2024 )
✓
✓
✗
✗
LiDAR-LLM ( Yang et al., 2025 )
✓
✓
✗
✗
MAPLM ( Cao et al., 2024 )
✓
✓
✗
✗
Table 1. Comparison of existing representative methods and KLDrive. ✓ indicates the property is supported and ✗ indicates it is not.
Figure 2. Overview of KLDrive, which converts multi-modal sensor data and a natural-language query into a scene KG, applies generic reasoning tools, and uses an in-context–learned LLM agent to produce a fact-grounded answer.
Figure 3. Energy-based scene KG generation model.
Figure 4. Proposed LLM Scene-Reasoning Agent.
Figure 5. Structured system prompt for the LLM planner.
Models
NuScenes-QA ( Qian et al., 2024 )
GVQA ( Sima et al., 2024 )
Acc ↑ (Exist)
Acc ↑ (Count)
Acc ↑ (Object)
Acc ↑ (Status)
Acc ↑ (Comparison)
Overall Acc ↑
SPICE ↑
METEOR ↑
CIDEr ↑
FocalFormer3D † ( Chen et al., 2023 )
50.28
0.01
0.55
0.22
47.36
24.02
14.41
0.21
0.56
RayDN † ( Liu et al., 2024a )
44.13
0.04
1.48
0.28
44.38
21.46
11.58
0.20
0.46
IS-Fusion † ( Yin et al., 2024 )
52.70
0.04
0.42
0.67
44.87
24.86
11.20
0.19
0.44
LLaVA-v1.6-mistral-7 ‡
48.64
6.83
1.48
5.09
52.80
26.17
43.75
0.11
0.38
Internvl-3.5-8b ‡
49.86
11.16
35.74
22.29
38.72
34.28
32.66
0.13
0.34
Table 2. Comparison for KLDrive with baselines on NuScenes-QA ( Qian et al., 2024 ) and GVQA ( Sima et al., 2024 ) . Bold numbers indicate the best performance. Methods marked with † trained since they have task-specific parameters, whereas methods marked with ‡ use frozen LLM with few-shot ICL only.
Figure 8
Exist
Count
Object
Status
Comparison
Overall Acc
w/o KG Generation Module
53.39
12.5
16.54
8.47
48.85
28.10
w/ KG Module (KLDrive-KG)
74.67
64.46
54.30
57.84
53.23
65.04
w/o LLM Agent Module
59.54
20.71
71.14
62.31
55.52
53.50
w/ LLM Agent Module (KLDrive-GT)
85.55
78.92
80.11
85.72
82.01
84.49
Table 3. Ablation comparison (accuracy%) of each functional module in KLDrive illustrating the impact of KG generation model and designed LLM agent.
Question answering studies how AI systems interpret questions and produce accurate, evidence-grounded answers across text, dialogue, and multimodal data.
Autonomous-driving question answering requires reasoning over structured scene information, yet existing vision-language approaches largely delegate heterogeneous reasoning operations to a single neural inference process. We argue that this uniform strategy overlooks a fundamental distinction: some queries admit exact symbolic solutions, while others require semantic interpretation. We introduce a query-adaptive neuro-symbolic reasoning framework that explicitly allocates computation according to the nature of the query. At its core is a hierarchical Spatiotemporal Scene Graph (STSG) that separates persistent object identities from frame-specific states and represents spatial relations and temporal transitions as explicit directed structures. Given a query, a symbolic executor first attempts to resolve it through exact graph operations; only when symbolic execution abstains is an LLM invoked for semantic reasoning. For these unresolved queries, query-conditioned graph retrieval and evidence filtering preserve relation direction, temporal locality, and object semantics, providing the LLM with compact and verified task-relevant evidence. This design shifts the role of the LLM from a universal reasoning engine to a targeted semantic reasoner, while allowing deterministic computation to be handled exactly and efficiently. We evaluate the framework on 5,916 NuScenes-QA questions across all ten scenes of nuScenes v1.0-mini under an oracle-perception setting. The complete system achieves 80.63 percent overall accuracy with GPT-5.4-mini, improving over the corresponding LLM-only configuration by 5.48 percentage points; with DeepSeek-V4-Flash, the improvement reaches 6.64 points. The largest gains occur on counting questions, with improvements of 10.20 and 12.61 points, respectively. These results show that selective reasoning improves both accuracy and inference efficiency.
Yiyao Wang, Pei Liu, Fangzhou Liu +1
The Hong Kong University of Science and Technology (Guangzhou) · Harbin Institute of Technology
Vision-language models have recently emerged as promising planners for autonomous driving, where success hinges on reasoning over spatial structure and dynamic interactions from multimodal input. However, existing models are typically trained without supervision that explicitly encodes relational dependencies, requiring them to infer interactions between traffic entities implicitly from raw sensor data. In this work, we introduce GraphPilot, a model-agnostic approach that conditions language-based driving models on structured relational context in the form of traffic scene graphs. We serialize scene graphs at different abstraction levels and formats and inject them through structured prompts without modifying the underlying architecture or training objective. Extensive closed-loop evaluations on LangAuto and Bench2Drive show persistent improvements across LMDrive, BEVDriver, and SimLingo. Controlled ablations and hidden-state probing further indicate that relational supervision contributes substantially to these improvements while increasing the decodability of scene relations. Open-loop SpaceDrive results on nuScenes further demonstrate transfer to real-world data. These results indicate that diverse architectures can effectively internalize and ground relational priors through scene graph-conditioned training, even without requiring scene graph input at test-time. Code, fine-tuned models, and our scene graph dataset are publicly available at https://github.com/iis-esslingen/GraphPilot.
Fabian Schmidt, Markus Enzweiler, Abhinav Valada
Esslingen University of Applied Sciences · University of Freiburg
We present Qwen-Drive-1.0, an initial step towards a vision-language foundation model for autonomous driving. Qwen-Drive-1.0 retains the architecture of the pretrained vision-language model (VLM) and integrates 3D perception, visual question answering, and motion planning within a unified framework. An external bird's-eye-view (BEV) perception head jointly performs 3D object detection, semantic occupancy prediction, and BEV map segmentation. It serves as a probe of the 3D information accessible from the shared representations and provides an explicit, inspectable interface to 3D scene structure. A Planning Expert conditions on shared VLM representations to generate future ego trajectories. A staged training recipe combines driving supervision with general-purpose vision-language data to acquire driving-specific competence while helping preserve broad visual understanding and instruction-following capabilities. Experiments demonstrate strong 3D perception and driving scene understanding while largely preserving general vision-language capability. Comprehensive evaluations across open-loop, pseudo-closed-loop, and closed-loop settings further show highly competitive motion-planning performance.
Xin Zhou, Zongchuang Zhao, Zhibo Yang +13
Qwen Team, Huazhong University of Science and Technology