UAV traffic scene understanding: A regulation embedded multi-modal network and a unified benchmark
Authors: Yu Zhang, Zhicheng Zhao, Ze Luo, Chenglong Li, Jin Tang
Organizations: Computer Network Information Center, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100190, China · Anhui Provincial Key Laboratory of Multi-modal Cognitive Computation, Anhui University, Hefei 230601, China · Information Materials and Intelligent Sensing Laboratory of Anhui Province, Anhui University, Hefei 230601, China
Traffic scene understanding from unmanned aerial vehicle (UAV) platforms is crucial for intelligent transportation systems due to its flexible deployment and wide-area monitoring capabilities. However, existing methods face significant challenges in real-world surveillance, as their heavy reliance on optical imagery leads to severe performance degradation under adverse illumination conditions like nighttime and fog. Furthermore, current Visual Question Answering (VQA) models are restricted to elementary perception tasks, lacking the domain-specific regulatory knowledge required to assess complex traffic behaviors. To address these limitations, we propose a novel Multi-modal Traffic Cognition Network (MTCNet) for robust UAV traffic scene understanding. Specifically, we design a Prototype-Guided Knowledge Embedding (PGKE) module that leverages high-level semantic prototypes from an external Traffic Regulation Memory (TRM) to anchor domain-specific knowledge into visual representations, enabling the model to comprehend complex behaviors and distinguish fine-grained traffic violations. Moreover, we develop a Quality-Aware Spectral Compensation (QASC) module that exploits the complementary characteristics of optical and thermal modalities to perform bidirectional context exchange, effectively compensating for degraded features to ensure robust representation in complex environments. In addition, we construct Traffic-VQA, the first large-scale optical-thermal infrared benchmark for cognitive UAV traffic understanding, comprising 8,180 aligned image pairs and 1.3 million question-answer pairs across 31 diverse types. Extensive experiments demonstrate that MTCNet significantly outperforms state-of-the-art methods in both cognition and perception scenarios. The dataset is available at https://github.com/YuZhang-2004/UAV-traffic-scene-understanding.
Figures & tables
Figure 1: Current challenges in UAV-based traffic VQA. (a) Data Gap. Existing datasets (top) rely on single-modal optical imagery for elementary perception, whereas practical surveillance (bottom) demands aligned OPT-TIR data for complex cognitive understanding. (b) Methodological Bottlenecks. General MLLMs struggle with the Domain Knowledge Gap , failing to interpret specific traffic rules ( e.g. , missing the "illegal" attribute of a U-turn), and the Multi-Modal Fusion Gap , where static fusion allows degraded optical noise to corrupt robust thermal features under adverse conditions. (c) Our Solution. The proposed MTCNet systematically bridges these gaps through the Traffic-VQA dataset, the PGKE module, and the QASC module.
Figure 2: Overall framework of MTCNet for multi-spectral UAV traffic VQA. The architecture adopts a Gated Parallel Residual paradigm in which the frozen, pre-trained MLLM visual features are adaptively augmented by domain-specific residual knowledge generated by the PGKE and QASC modules. The learnable gating parameters α and β regulate the intensity of cognitive and multi-modal context injection.
Figure 3: Multi-modal visual grounding in the TRM construction pipeline. Grounding prompts generated from semantic phrase distillation are used to localize traffic entities and behaviors (e.g., linear walkways, vehicle turning). Red boxes indicate the extracted regions of interest, demonstrating accurate text-to-region alignment in both optical and thermal imagery.
Figure 4: Internal architecture of the PGKE module. The module performs question-guided similarity retrieval to identify the top- K most relevant prototypes from the TRM. These prototypes serve as keys and values in a Multi-Head Cross-Attention mechanism, injecting situational domain knowledge into the visual feature streams as an optimized residual increment ΔFPGKE .
Figure 5: Illustrative examples from the Traffic-VQA dataset. (a) Synchronized and co-registered optical and TIR UAV image pairs across diverse urban traffic settings. (b) Examples of challenging cognitive question-answer pairs that require deep situational understanding, such as identifying traffic violations and inferring latent behavioral risks.
Figure 6: Sample aligned image pairs from the proposed Traffic-VQA dataset. The dataset spans diverse illumination conditions (e.g., daylight, low-light, and nighttime) alongside adverse weather scenarios (e.g., fog), providing well-aligned OPT and TIR images to support research in all-weather traffic understanding.
Figure 7: Statistical distribution of the Traffic-VQA dataset. Left: Breakdown of the 8,180 image pairs by illumination conditions, weather, and scene type. Right: Distribution of the 31 question types, highlighting the significant proportion of high-level cognitive tasks (20.7%) and specialized multi-modal queries.
Figure 8: Distribution of average question length (in words) across varying question types within the Traffic-VQA dataset.
Figure 9: Representative question-answer samples from Traffic-VQA. The dataset covers diverse query types beyond object recognition, requiring reasoning over spatial relations, vehicle behaviors, traffic-rule compliance, and scene context (e.g., queuing, environment type).
Statistics
Numbers
Total QAs
1,301,466
Total OPT-TIR pairs
8,180
Avg QA on img pair
159.1
Max question length
28
Avg question length
8.5
Question vocab size
24,357
Table 2: Categorization of the 31 question types in the Traffic-VQA dataset, divided into cognitive, perceptual, and multi-modal domains.
Model
Source
OA
Traffic Cognition
Traffic Perception
Veh. Violation
Veh. Behavior
Veh.
Ped. Violation
Ped. Behavior
Ped.
OPT modality
MiniCPM-V [ 44 ]
arXiv 2024
45.27
39.7
45.43
37.12
39.57
47.57
40.13
44.82
41.22
MiniGPT-v2 [ 45 ]
arXiv 2023
36.04
35.3
35.95
33.74
37.38
37.96
29.54
39.05
40.14
DeepSeek-VL [ 12 ]
arXiv 2024
42.64
38.62
41.51
37.01
37.79
44.38
41.23
42.92
34.86
GeoPix [ 46 ]
GRSM 2025
43.59
38.13
40.48
37.16
39.49
45.1
38.07
43.94
41.86
GeoChat [ 3 ]
CVPR 2024
44
41.24
44.39
41.98
41.38
45.32
42.97
42.56
46.05
Table 4: Quantitative comparison on the Traffic-VQA test set. All numbers are Acc (%). Bold indicates the best result and underline indicates the second-best result within the same modality for each metric.
ID
PGKE (Cognition)
QASC (Perception)
OA (%)
AA (%)
Exp1
✗
✗
47.62
46.93
Exp2
✓
✗
60.67
59.41
Exp3
✗
✓
61.27
59.86
Exp4
✓
✓
61.94
59.97
Table 5: Ablation study isolating the PGKE and QASC modules under the few-shot setting. Example 1 represents the unmodified baseline, where features are aggregated element-wise. OA : Overall Accuracy, AA : Average Accuracy.
Integration Method
Overall Accuracy (OA)
Δ vs. Baseline
Optical Only (Baseline)
47.56
-
Thermal Only
44.09
-
Element-wise Add
47.62
+0.06
Concatenation
49.12
+1.56
MTCNet (QASC)
61.94
+14.38
Table 6: Comparison of different modality integration methods. The proposed QASC module consistently outperforms all static fusion operations.
Figure 10: Performance comparison (quantified via CIDEr score) across the 31 question types in Traffic-VQA. The radar chart highlights the operational stability of the proposed multi-modal (MUL) approach, particularly in challenging environmental categories such as "Night" and "Mist," where single-modality approaches consistently underperform.
Remote sensing visual question answering (VQA) studies how AI answers natural-language questions about satellite, aerial, and other Earth-observation imagery.
Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Institute of Artificial Intelligence (TeleAI), China Telecom +2
Dalian Maritime University, Dalian, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Hubei University of Economics, Wuhan, China