Multi-Modal Graph Neural Network-Inspired Vessel Identification and Tracking for UAV-Based Maritime Surveillance

Chenjie Zhao; Jiangtian Nie; Jiawen Kang; Yuxu Lu; Ryan Wen Liu

cs.CVOct 9, 2026

Multi-Modal Graph Neural Network-Inspired Vessel Identification and Tracking for UAV-Based Maritime Surveillance

Authors: Chenjie Zhao, Jiangtian Nie, Jiawen Kang, Yuxu Lu, Ryan Wen Liu

Organizations: Sanya Science and Education Innovation Park of Wuhan University of Technology, Sanya 572025, China · School of Navigation, Wuhan University of Technology, Wuhan 430063, China · School of Natural and Computing Sciences, University of Aberdeen, Aberdeen AB24 3UE, U.K. · School of Automation, Guangdong University of Technology, Guangzhou 510006, China ·

Abstract

The unmanned aerial vehicle (UAV)-based maritime surveillance is becoming an important part of maritime Internet of Things (IoT) and intelligent transportation systems. It provides a flexible and cost-effective way to large-scale maritime traffic monitoring. However, achieving robust situational awareness remains challenging in practical maritime environments. The main challenges arise from heterogeneous sensing modalities, adverse imaging conditions, and spatial misalignment caused by UAV motion. To address these problems, this paper proposes an airborne multi-modal graph learning-driven framework, termed AMGT, for vessel identification and tracking in UAV-assisted maritime IoT. Specifically, visible and infrared images are first aligned and fused through a multi-parameter perspective transformation model. This improves visual perception under low visibility and complex illumination conditions. Automatic identification system (AIS) messages are then temporally synchronized using interpolation and linear prediction. To detect small and oriented vessels, we develop a mixed-modality MAir-YOLOv8-OBB detector, followed by the MAir-SORT algorithm for stable vessel tracking. We further design a graph learning-driven association network to model temporal dynamics and spatial dependencies among vessels and between vessels and UAVs. A motion-aware coordinate unification (MCU) strategy is also introduced to reduce nonlinear misalignment caused by UAV motion. Experiments on the MAP-M2UAV50 dataset show that AMGT achieves high accuracy and efficiency. The results demonstrate its potential for UAV-assisted maritime IoT applications, including vessel monitoring, traffic situational awareness, and intelligent maritime transportation services. The dataset and code will be available at https://github.com/MAPGroup/AMGT.

Figures & tables

Fig. 1: The low-altitude sensing, communication, computing, and control (SCCC) system for UAV-assisted maritime IoT. The system consists of multiple maritime UAVs (M-UAVs), takeoff and landing platforms, ground stations, and a central controller connected by air-ground communication links. With the aid of deep learning-based models and robust algorithms, M-UAVs can support real-time maritime surveillance and collaborative decision-making in complex marine environments.

UAV Aerial Dataset	Maritime Scenario	Perception Task	Images	Videos	Data Modal	Total Anno.	Num. of Class	Altitude (m)	Oriented RBbox
Seagull [ 25 ]	Sea Monitoring & Surveillance	Object Detection & Tracking	——	19	Visible Infrared	——	6	150-300	$\times$
AFO [ 26 ]	Search & Rescue	Object Detection	3640	50	Visible	12K	9	30-80	$\times$
MOBDrone [ 27 ]	Search & Rescue	Object Detection	125K	66	Visible	180K	5	10-60	$\times$
Zhao et al. [ 28 ]	Maritime Surveillance	Object Detection	18K	3	Visible	——	1	120	$\times$
MAP-M2UAV50	Maritime Surveillance	Object Detection, Tracking & Association	12K	50	Visible Infrared AIS & Flight data	70K	3	150-300	✓

TABLE I: The UAV aerial datasets in maritime scenarios. Our dataset comprises data from four distinct modalities. The rotated bounding boxes (RBboxes) were manually annotated. Additionally, various weather conditions, including sunny, hazy, night, and sun glare scenarios, are encompassed.

Fig. 2: The overall architecture of our AMGT, a multi-modal graph learning-driven multi-vessel identification and tracking framework. Four data modalities are input, i.e., visible and IR videos, AIS, and UAV flight data. The data preprocessing mainly includes image fusion under adverse weather, as well as AIS decoding, cleaning, and prediction. Detection is performed by an enhanced MAir‑YOLOv8‑OBB network, followed by MAir-SORT vessel tracking method. Visual and AIS positions are spatially aligned and normalized to form two graphs, which are then associated using a spatio‑temporal graph neural network. Tracking IDs are ultimately corrected to output identified and tracked vessels.

Fig. 3: From top to bottom: flow diagram of visible and IR image fusion method, the input original low visibility images, and the output clear fused images. Four challenges are presented: (a) Hazy, (b) Night, (c)Sun Glare, (d) Tiny Objects.

Fig. 4: The diagram of our MAir-YOLOv8-OBB detection network. The input comprises UAV aerial imagery across varying weather and water conditions. Image features are extracted and fused via the backbone and enhanced neck, followed by the prediction of rotated bounding boxes through the detection head.

Fig. 5: The illustration of attitude compensation for UAV motion. (a) Pitch compensation in the side view. (b) Yaw compensation and relative ENU projection in the top view.

Fig. 6: The motion-aware coordinate unification (MCU) across three UAV flight scenarios: (a) Rapid Motion (Flight 1), (b) Hovering (Flight 2), and (c) Slow Motion (Flight 3). The top displays the trajectories of visual and AIS objects projected onto the motion-compensated metric plane, and the bottom depicts UAV position and velocity variations. The rough alignment of trajectories in all cases demonstrates that the unification method is robust and insensitive to complex maneuvers of the UAV.

Detection Methods	Precision ( $\%$ ) $\uparrow$	Recall ( $\%$ ) $\uparrow$	mAP50 ( $\%$ ) $\uparrow$	mAP50-95 ( $\%$ ) $\uparrow$	Params (M) $\downarrow$	GFLOPS $\downarrow$	Time (ms) $\downarrow$
YOLOv8n-OBB	94.62	84.21	93.04	77.11	0 3.08	0 8.44	2.37
YOLOv8s-OBB	90.59	88.65	93.16	78.65	11.42	29.63	2.37
YOLOv11n-OBB	94.63	84.29	93.09	77.08	0 2.66	0 6.69	2.91
YOLOv11s-OBB	93.65	86.55	93.11	78.54	0 9.71	22.53	3.12
MAir-YOLOv8n-OBB	96.58	94.00	97.77	80.81	0 2.92	12.10	2.68
MAir-YOLOv8s-OBB	97.23	93.55	97.81	82.00	10.62	35.93	2.58

TABLE II: The quantitative comparison of MAir-YOLO and baseline detectors on the 12K-image subset of the MAP-M2UAV50 dataset. The suffixes n and s denote the nano and small model variants, respectively, and OBB denotes oriented bounding box. MAir-YOLOv8s-OBB is selected as the detector in AMGT based on its accuracy–efficiency trade-off.

Tracking Methods	IDS $\downarrow$	Precision ( $\%$ ) $\uparrow$	Recall ( $\%$ ) $\uparrow$	MOTA ( $\%$ ) $\uparrow$	MOTP ( $\%$ ) $\uparrow$	IDF1 ( $\%$ ) $\uparrow$	HOTA ( $\%$ ) $\uparrow$	Time (ms) $\downarrow$
SORT [ 15 ] [ 20 ]	56	92.31	89.94	81.97	78.28	79.24	65.91	0.182
ByteTrack [ 16 ]	63	89.46	90.05	78.90	77.67	78.87	64.87	0.718
BoT-SORT [ 17 ]	12	90.05	92.31	82.00	79.27	89.51	71.36	10.801
MAir-SORT	2	92.80	90.85	83.78	79.53	91.76	72.77	2.874

TABLE III: The overall tracking performance on eight representative airborne maritime video sequences using MAir-YOLOv8s-OBB detection results. Time denotes the average tracking update time per frame.

Fig. 7: The qualitative comparison of SORT [ 15 ] [ 20 ] , ByteTrack [ 16 ] , BoT-SORT [ 17 ] , and MAir-SORT under bridge occlusion, sun glare, nighttime imaging, and haze. Columns correspond to tracking methods, and rows correspond to scene conditions. Each method is shown at two frames from the same sequence. The ID pair below the images gives the identities assigned to the same selected vessel at the two frames. Green text indicates an unchanged identity, while red text indicates an identity switch. The tracked IDs are generated independently by each method.

Fig. 8: The training loss convergence of the six learning-based association methods over 100 epochs. Curves and shaded regions denote the mean and sample standard deviation over three random seeds. The inset panels enlarge epochs 35-50 and 85-100.

Association Methods	Precision ( $\%$ ) $\uparrow$	Recall ( $\%$ ) $\uparrow$	F1 ( $\%$ ) $\uparrow$	A-IDS $\downarrow$	Time (ms/window) $\downarrow$
CD [ 35 ]	14.49	11.81	13.02	17	0.198
DTW [ 36 ]	15.94	12.99	14.32	17	5.61
ED [ 37 ]	15.94	12.99	14.32	17	0.081
LSTM [ 21 ]	84.36 ± 3.09	82.81 ± 2.85	83.58 ± 2.97	9.00 ± 1.00	0.721 ± 0.003
BiGRU [ 38 ]	93.69 ± 1.50	89.24 ± 4.21	91.37 ± 2.14	8.67 ± 0.58	0.514 ± 0.001
GRU [ 22 ]	91.73 ± 2.27	90.29 ± 2.24	91.01 ± 2.26	13.00 ± 3.46	0.514 ± 0.001

TABLE IV: The quantitative visual-AIS association comparison with and without motion-aware coordinate unification (MCU). All methods use the same MAir-YOLOv8s-OBB detections and MAir-SORT outputs. Learning-based methods report the mean

\pm

sample standard deviation over three random seeds.

Fig. 9: The visualization of the trajectory association under varying traffic conditions. The maritime scenarios are broadly classified into: (a) Low-density and (b) High-density. The green dashed lines (predicted associations) accurately connect the visual trajectories (red lines) with their corresponding AIS trajectories (blue lines). The low-density scenarios have clear spatial layouts, while high-density scenarios involve intricate inter-vessel interactions and overlaps.

Fig. 10: The visualization of multi-vessel identification and tracking under three UAV flight motions (From top to bottom: Flight 1, 2, and 3, which are shown in Fig. 6 . From left to right: (a) and (b) Results at time T1 and T2, (c) Trajectory matching by prediction (left) and ground truth (GT, right), (d) Score heatmap by prediction (left) and GT (right).

Maritime Surveillance

See all 12 papers

CmIVTP: Cross-modal Interaction-based Vessel Trajectory Prediction for Maritime Intelligence
May 26, 2026Yuxu Lu, Dong Yang, Xiaoyu Li +2Intelligent Transportation Systems Vessel Trajectory Prediction
Department of Logistics and Maritime Studies, the Hong Kong Polytechnic University, Hong Kong 999077 · Department of Logistics and Maritime Studies and the Research Centre for ESG Advancement (RCESGA), the Hong Kong Polytechnic University, Hong Kong 999077 · Department of Logistics and Maritime Studies, the Hong Kong Polytechnic University, Hong Kong 999077, and also with the School of Navigation, Wuhan University of Technology, Wuhan 430063
HDST-GNN: Heterogeneous Dynamic Spatiotemporal Graph Neural Networks for Multi-Object Tracking in UAV Aerial Imagery
Jun 4, 2026Phillip JiangHeterogeneous GNNs Visual Object Tracking
Appsofa LLC
Alignment-Aware and Reliability-Gated Multimodal Fusion for Unmanned Aerial Vehicle Detection Across Heterogeneous Thermal-Visual Sensors
Mar 9, 2026Ishrat Jahan, Molla E Majid, M Murugappan +5UAV Detection Multimodal Sensor Fusion
Department of Computer Science and Engineering, Shahjalal University of Science & Technology, Sylhet, Bangladesh · Computer Applications Department, Academic Bridge Program, Qatar Foundation, Doha, Qatar. · Intelligent Signal Processing (ISP) Research Lab, Department of Electronics and Communication Engineering, Kuwait College of Science and Technology, Block 4, Doha, Kuwait +5

May 26, 2026J/K move · Enter open · S save

CmIVTP: Cross-modal Interaction-based Vessel Trajectory Prediction for Maritime Intelligence

Yuxu Lu, Dong Yang, Xiaoyu Li, Mengwei Bao+1

Department of Logistics and Maritime Studies, the Hong Kong Polytechnic University, Hong Kong 999077 · Department of Logistics and Maritime Studies and the Research Centre for ESG Advancement (RCESGA), the Hong Kong Polytechnic University, Hong Kong 999077 · Department of Logistics and Maritime Studies, the Hong Kong Polytechnic University, Hong Kong 999077, and also with the School of Navigation, Wuhan University of Technology, Wuhan 430063

Intelligent Transportation Systems Vessel Trajectory Prediction Multimodal Trajectory Prediction Trajectory Prediction Multimodal Sensor Fusion

Open paper AskPDF arXiv

Maritime intelligent transportation systems (MITS) are essential for ensuring navigation safety and efficiency in busy waterways. However, accurate vessel trajectory prediction remains challenging due to the limitations of single-source data. Automatic identification system (AIS) data is often sparse or unavailable for small vessels, while closed-circuit television (CCTV) data alone cannot fully capture dynamic vessel behavior. To mitigate these challenges, we propose a cross-modal interaction-based vessel trajectory prediction (named CmIVTP) framework to model the intricate interactions between vessel dynamics and environmental constraints. Specifically, we introduce a target-aware scene encoder to extract scene semantic features, effectively capturing vessel-environment interactions and enhancing trajectory prediction accuracy. In addition, we propose a cross-modal interaction transformer, which integrates AIS-derived motion features, CCTV-based environmental features, and scene representations. It leverages cross-modal attention mechanisms to simultaneously capture intra-modal semantics and inter-modal interactions, ensuring dynamically consistent and environmentally feasible predictions. Furthermore, we construct a vessel group trajectory bank by clustering historical AIS trajectories into representative motion patterns, providing an efficient and scalable approach for candidate trajectory generation. Additionally, we introduce the maritime multimodal dataset plus (named Maritime-MmD

^+

), a large-scale dataset that synchronizes AIS data and CCTV video data, providing robust support for multimodal trajectory prediction research. Extensive experiments demonstrate that CmIVTP achieves better performance on multimodal-driven vessel trajectory prediction benchmarks. The code resources for this work can be available at https://github.com/LouisYxLu/CmIVTP.

Abstract

Maritime Surveillance

Explore similar work

CmIVTP: Cross-modal Interaction-based Vessel Trajectory Prediction for Maritime Intelligence

HDST-GNN: Heterogeneous Dynamic Spatiotemporal Graph Neural Networks for Multi-Object Tracking in UAV Aerial Imagery

Alignment-Aware and Reliability-Gated Multimodal Fusion for Unmanned Aerial Vehicle Detection Across Heterogeneous Thermal-Visual Sensors