UAV traffic scene understanding: A regulation embedded multi-modal network and a unified benchmark
Authors: Yu Zhang, Zhicheng Zhao, Ze Luo, Chenglong Li, Jin Tang
Organizations: Computer Network Information Center, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100190, China · Anhui Provincial Key Laboratory of Multi-modal Cognitive Computation, Anhui University, Hefei 230601, China · Information Materials and Intelligent Sensing Laboratory of Anhui Province, Anhui University, Hefei 230601, China
Traffic scene understanding from unmanned aerial vehicle (UAV) platforms is crucial for intelligent transportation systems due to its flexible deployment and wide-area monitoring capabilities. However, existing methods face significant challenges in real-world surveillance, as their heavy reliance on optical imagery leads to severe performance degradation under adverse illumination conditions like nighttime and fog. Furthermore, current Visual Question Answering (VQA) models are restricted to elementary perception tasks, lacking the domain-specific regulatory knowledge required to assess complex traffic behaviors. To address these limitations, we propose a novel Multi-modal Traffic Cognition Network (MTCNet) for robust UAV traffic scene understanding. Specifically, we design a Prototype-Guided Knowledge Embedding (PGKE) module that leverages high-level semantic prototypes from an external Traffic Regulation Memory (TRM) to anchor domain-specific knowledge into visual representations, enabling the model to comprehend complex behaviors and distinguish fine-grained traffic violations. Moreover, we develop a Quality-Aware Spectral Compensation (QASC) module that exploits the complementary characteristics of optical and thermal modalities to perform bidirectional context exchange, effectively compensating for degraded features to ensure robust representation in complex environments. In addition, we construct Traffic-VQA, the first large-scale optical-thermal infrared benchmark for cognitive UAV traffic understanding, comprising 8,180 aligned image pairs and 1.3 million question-answer pairs across 31 diverse types. Extensive experiments demonstrate that MTCNet significantly outperforms state-of-the-art methods in both cognition and perception scenarios. The dataset is available at https://github.com/YuZhang-2004/UAV-traffic-scene-understanding.
Figures & tables
Figure 1: Current challenges in UAV-based traffic VQA. (a) Data Gap. Existing datasets (top) rely on single-modal optical imagery for elementary perception, whereas practical surveillance (bottom) demands aligned OPT-TIR data for complex cognitive understanding. (b) Methodological Bottlenecks. General MLLMs struggle with the Domain Knowledge Gap , failing to interpret specific traffic rules ( e.g. , missing the "illegal" attribute of a U-turn), and the Multi-Modal Fusion Gap , where static fusion allows degraded optical noise to corrupt robust thermal features under adverse conditions. (c) Our Solution. The proposed MTCNet systematically bridges these gaps through the Traffic-VQA dataset, the PGKE module, and the QASC module.
Figure 2: Overall framework of MTCNet for multi-spectral UAV traffic VQA. The architecture adopts a Gated Parallel Residual paradigm in which the frozen, pre-trained MLLM visual features are adaptively augmented by domain-specific residual knowledge generated by the PGKE and QASC modules. The learnable gating parameters α and β regulate the intensity of cognitive and multi-modal context injection.
Figure 3: Multi-modal visual grounding in the TRM construction pipeline. Grounding prompts generated from semantic phrase distillation are used to localize traffic entities and behaviors (e.g., linear walkways, vehicle turning). Red boxes indicate the extracted regions of interest, demonstrating accurate text-to-region alignment in both optical and thermal imagery.
Figure 4: Internal architecture of the PGKE module. The module performs question-guided similarity retrieval to identify the top- K most relevant prototypes from the TRM. These prototypes serve as keys and values in a Multi-Head Cross-Attention mechanism, injecting situational domain knowledge into the visual feature streams as an optimized residual increment ΔFPGKE .
Figure 5: Illustrative examples from the Traffic-VQA dataset. (a) Synchronized and co-registered optical and TIR UAV image pairs across diverse urban traffic settings. (b) Examples of challenging cognitive question-answer pairs that require deep situational understanding, such as identifying traffic violations and inferring latent behavioral risks.
Figure 6: Sample aligned image pairs from the proposed Traffic-VQA dataset. The dataset spans diverse illumination conditions (e.g., daylight, low-light, and nighttime) alongside adverse weather scenarios (e.g., fog), providing well-aligned OPT and TIR images to support research in all-weather traffic understanding.
Figure 7: Statistical distribution of the Traffic-VQA dataset. Left: Breakdown of the 8,180 image pairs by illumination conditions, weather, and scene type. Right: Distribution of the 31 question types, highlighting the significant proportion of high-level cognitive tasks (20.7%) and specialized multi-modal queries.
Figure 8: Distribution of average question length (in words) across varying question types within the Traffic-VQA dataset.
Figure 9: Representative question-answer samples from Traffic-VQA. The dataset covers diverse query types beyond object recognition, requiring reasoning over spatial relations, vehicle behaviors, traffic-rule compliance, and scene context (e.g., queuing, environment type).
Statistics
Numbers
Total QAs
1,301,466
Total OPT-TIR pairs
8,180
Avg QA on img pair
159.1
Max question length
28
Avg question length
8.5
Question vocab size
24,357
Table 2: Categorization of the 31 question types in the Traffic-VQA dataset, divided into cognitive, perceptual, and multi-modal domains.
Model
Source
OA
Traffic Cognition
Traffic Perception
Veh. Violation
Veh. Behavior
Veh.
Ped. Violation
Ped. Behavior
Ped.
OPT modality
MiniCPM-V [ 44 ]
arXiv 2024
45.27
39.7
45.43
37.12
39.57
47.57
40.13
44.82
41.22
MiniGPT-v2 [ 45 ]
arXiv 2023
36.04
35.3
35.95
33.74
37.38
37.96
29.54
39.05
40.14
DeepSeek-VL [ 12 ]
arXiv 2024
42.64
38.62
41.51
37.01
37.79
44.38
41.23
42.92
34.86
GeoPix [ 46 ]
GRSM 2025
43.59
38.13
40.48
37.16
39.49
45.1
38.07
43.94
41.86
GeoChat [ 3 ]
CVPR 2024
44
41.24
44.39
41.98
41.38
45.32
42.97
42.56
46.05
Table 4: Quantitative comparison on the Traffic-VQA test set. All numbers are Acc (%). Bold indicates the best result and underline indicates the second-best result within the same modality for each metric.
ID
PGKE (Cognition)
QASC (Perception)
OA (%)
AA (%)
Exp1
✗
✗
47.62
46.93
Exp2
✓
✗
60.67
59.41
Exp3
✗
✓
61.27
59.86
Exp4
✓
✓
61.94
59.97
Table 5: Ablation study isolating the PGKE and QASC modules under the few-shot setting. Example 1 represents the unmodified baseline, where features are aggregated element-wise. OA : Overall Accuracy, AA : Average Accuracy.
Integration Method
Overall Accuracy (OA)
Δ vs. Baseline
Optical Only (Baseline)
47.56
-
Thermal Only
44.09
-
Element-wise Add
47.62
+0.06
Concatenation
49.12
+1.56
MTCNet (QASC)
61.94
+14.38
Table 6: Comparison of different modality integration methods. The proposed QASC module consistently outperforms all static fusion operations.
Figure 10: Performance comparison (quantified via CIDEr score) across the 31 question types in Traffic-VQA. The radar chart highlights the operational stability of the proposed multi-modal (MUL) approach, particularly in challenging environmental categories such as "Night" and "Mist," where single-modality approaches consistently underperform.
Remote sensing visual question answering (VQA) studies how AI answers natural-language questions about satellite, aerial, and other Earth-observation imagery.
In Intelligent Transportation System (ITS), unmanned aerial vehicle (UAV)-based surveillance offers an innovative solution to traffic surveillance with wide coverage and real-time data collection capabilities. In comparison to fixed ground-based infrastructure, UAVs are able to respond to dynamic traffic but present challenges such as vehicle detection at varying altitudes, compensation for motion-induced image variations and efficient processing of high-resolution images. Deep learning has been largely beneficial on improving the detection accuracy; however, for practical deployment, a critical assessment of the accuracy, latency, and harmonization with current transportation systems needs to be carefully considered. This survey reviews recent advancements in the UAV-based traffic monitoring, with a primary focus being deep neural network models for traffic analytics in various urban settings. Three main challenges identified in the literature are ensuring compatibility with traffic control systems, achieving real-time processing to optimize traffic flow, and maintaining robust detection in different environmental conditions. Existing solutions often lack comprehensive frameworks for utilizing UAV captured data to respond to incidents and manage traffic effectively. Future research should focus on optimal detection models, edge processing, and adaptive control integration to improve the responsiveness of urban traffic management.
Jianlin Ye, Christos Kyrkou
The authors are with the KIOS Research and Innovation Centre of Excellence (KIOS CoE), and University of Cyprus, Nicosia, 1678, Cyprus.
Unmanned Aerial Vehicle (UAV) videos are widely used in traffic monitoring, urban management, and emergency rescue. However, existing UAV video perception mainly relies on box-level localization and trajectory association under predefined categories, making it difficult to simultaneously support flexible queries and fine-grained instance-level dynamic understanding in open scenarios. To this end, we introduce a new task, UAV Open-Vocabulary Video Instance Segmentation (UAV-OVVIS), which discovers targets in UAV videos according to open-vocabulary queries and outputs instance-level segmentation trajectories with globally consistent identities. Considering the scarcity of instance-level annotations in UAV scenarios, we propose AeroTrack, a training-free unified framework. AeroTrack centers on periodic open-vocabulary detection, short-segment mask propagation, and cross-segment identity unification, reusing existing visual foundation models to enable UAV-OVVIS. Based on this framework, we instantiate five AeroTrack variants and construct AeroVIS, an evaluation benchmark for UAV-OVVIS containing 9 UAV object categories and 8,279 trajectories. Experiments show that AeroTrack substantially outperforms existing general video instance segmentation methods in UAV scenarios and demonstrates strong open-vocabulary robustness and generalization. To support future research, we release AeroTrack and AeroVIS as a unified framework and benchmark for UAV-OVVIS.
Mingyu Dou, Shi Qiu, Ming Hu +2
Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Institute of Artificial Intelligence (TeleAI), China Telecom +2
Low-light UAV-based RGB-infrared oriented small-vehicle detection is important for nighttime traffic monitoring, emergency response, and urban inspection. Illumination variations, headlight glare, local shadows, and thermal-response degradation cause spatially varying modality reliability, while the small visual extent of vehicles further weakens boundaries, orientation cues, and thermal responses. Accordingly, selecting trustworthy observations based on local modality reliability while further exploiting complementary discriminative information in regions with ambiguous modality preference is key to constructing effective multimodal representations. Based on this insight, we propose ReDiffNet, a reliability-conditioned differential representation network in which modality reliability guides both evidence selection and complementary recovery. Specifically, degradation-aware reliability learning estimates relative spatial reliability, uncertainty-guided differential recovery exploits cross-modal differences to recover complementary cues in ambiguous regions, and reliability-conditioned reconstruction integrates retained and recovered evidence into a unified representation. ReDiffNet achieves 85.3% and 73.9% mAP50 on DroneVehicle and VEDAI, respectively, supporting its effectiveness.
Qifan Zhang, Ziran Zhou, Ruijie Li +4
Dalian Maritime University, Dalian, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Hubei University of Economics, Wuhan, China