Navigational Safety of Marine Surface Vehicles in Restricted Visibility: From Intelligent Vision to Vision-Centered Multi-Sensor Fusion

With the rapid development of sensor and artificial intelligence (AI) technologies, intelligent surface vehicles (ISVs) have gained increasing attention from academia and industry. Their intelligence, reliability, and safety depend heavily on situational awareness in complex navigational environments. To achieve high-quality perception, we develop a vision-centered multi-sensor fusion system, named NavEYE, by exploiting complementary sensors, including the automatic identification system (AIS), radar, and RGB camera. Specifically, we first propose a multi-constrained gated data association method (MCGA) to accurately match low-temporal-resolution AIS data with high-temporal-resolution radar data. Their fusion result is then obtained by selectively implementing distance-aware adaptively weighted fusion (DAWF) and timeliness decay-based stitching fusion (TDSF), which reduce the uncertainty caused by AIS or radar data loss in real-world sensing scenarios. Based on accurate and robust visual object detection, we further associate and fuse AIS, radar, and visual data through joint constraints of normalized bearing and distance features. According to the fusion results, comprehensive information related to ships of interest can be automatically obtained, helping enhance situational awareness and reduce collision risk for ISVs. The feasibility, robustness, usability, and effectiveness of the proposed multi-sensor fusion method and situational awareness system are demonstrated through extensive experiments on a real-world sensing dataset collected from AIS, radar, and camera. The experimental results show the superior performance of our fusion method in both quantitative and qualitative evaluations. In addition, the shipboard NavEYE system can promote navigational safety for ISVs in complex and dynamic environments.

Marine scene understanding and segmentation plays a vital role in maritime monitoring and navigation safety. However, prevalent factors like fog and strong reflections in maritime environments cause severe image degradation, significantly compromising the stability of semantic perception. Existing restoration and enhancement methods typically target specific degradations or focus solely on visual quality, lacking end-to-end collaborative mechanisms that simultaneously improve structural recovery and semantic effectiveness. Moreover, publicly available infrared-visible datasets are predominantly collected from urban scenes, failing to capture the authentic characteristics of coupled degradations in marine environments. To address these challenges, the Infrared-Visible Maritime Ship Dataset (IVMSD) is proposed to cover various maritime scenarios under diverse weather and illumination conditions. Building upon this dataset, a Multi-task Complementary Learning Framework (MCLF) is proposed to collaboratively perform image restoration, multimodal fusion, and semantic segmentation within a unified architecture. The framework includes a Frequency-Spatial Enhancement Complementary (FSEC) module for degradation suppression and structural enhancement, a Semantic-Visual Consistency Attention (SVCA) module for semantic-consistent guidance, and a cross-modality guided attention mechanism for selective fusion. Experimental results on IVMSD demonstrate that the proposed method achieves state-of-the-art segmentation performance, significantly enhancing robustness and perceptual quality under complex maritime conditions.

The unmanned aerial vehicle (UAV)-based maritime surveillance is becoming an important part of maritime Internet of Things (IoT) and intelligent transportation systems. It provides a flexible and cost-effective way to large-scale maritime traffic monitoring. However, achieving robust situational awareness remains challenging in practical maritime environments. The main challenges arise from heterogeneous sensing modalities, adverse imaging conditions, and spatial misalignment caused by UAV motion. To address these problems, this paper proposes an airborne multi-modal graph learning-driven framework, termed AMGT, for vessel identification and tracking in UAV-assisted maritime IoT. Specifically, visible and infrared images are first aligned and fused through a multi-parameter perspective transformation model. This improves visual perception under low visibility and complex illumination conditions. Automatic identification system (AIS) messages are then temporally synchronized using interpolation and linear prediction. To detect small and oriented vessels, we develop a mixed-modality MAir-YOLOv8-OBB detector, followed by the MAir-SORT algorithm for stable vessel tracking. We further design a graph learning-driven association network to model temporal dynamics and spatial dependencies among vessels and between vessels and UAVs. A motion-aware coordinate unification (MCU) strategy is also introduced to reduce nonlinear misalignment caused by UAV motion. Experiments on the MAP-M2UAV50 dataset show that AMGT achieves high accuracy and efficiency. The results demonstrate its potential for UAV-assisted maritime IoT applications, including vessel monitoring, traffic situational awareness, and intelligent maritime transportation services. The dataset and code will be available at https://github.com/MAPGroup/AMGT.