Organizations: Laboratory of Underwater Acoustic Communication and Marine Information Technology, Ministry of Education, Xiamen University, Xiamen, Fujian 361005, China. · PAMI Research Group, Department of Computer and Information Science, University of Macau, Macau, China · School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen 518107, China
Marine scene understanding and segmentation plays a vital role in maritime monitoring and navigation safety. However, prevalent factors like fog and strong reflections in maritime environments cause severe image degradation, significantly compromising the stability of semantic perception. Existing restoration and enhancement methods typically target specific degradations or focus solely on visual quality, lacking end-to-end collaborative mechanisms that simultaneously improve structural recovery and semantic effectiveness. Moreover, publicly available infrared-visible datasets are predominantly collected from urban scenes, failing to capture the authentic characteristics of coupled degradations in marine environments. To address these challenges, the Infrared-Visible Maritime Ship Dataset (IVMSD) is proposed to cover various maritime scenarios under diverse weather and illumination conditions. Building upon this dataset, a Multi-task Complementary Learning Framework (MCLF) is proposed to collaboratively perform image restoration, multimodal fusion, and semantic segmentation within a unified architecture. The framework includes a Frequency-Spatial Enhancement Complementary (FSEC) module for degradation suppression and structural enhancement, a Semantic-Visual Consistency Attention (SVCA) module for semantic-consistent guidance, and a cross-modality guided attention mechanism for selective fusion. Experimental results on IVMSD demonstrate that the proposed method achieves state-of-the-art segmentation performance, significantly enhancing robustness and perceptual quality under complex maritime conditions.
Figures & tables
Fig. 1: Under complex maritime conditions, single-modality features exhibit limited discriminability due to severe degradation and modality-specific information loss. In contrast, the proposed method explicitly models degradation patterns and performs multimodal feature fusion, thereby substantially enhancing the robustness and discriminative capability of the learned representations.
Fig. 2: Examples of different scenes included in the IVMSD dataset.
Fig. 3: The overview of the proposed Multi-task Complementary Learning Framework.
IVMSD
Cargo Ship
Fishing Boat
Sand Dredger
Speedboat
mIoU
SeAFusion
21.1
0.0
3.2
0.0
24.6
SegFormer
47.0
24.5
82.2
22.3
55.0
LASNet
40.0
10.2
49.2
0.0
37.9
MDRNet+
23.2
0.01
67.0
10.1
40.4
SGFNet
28.5
0.1
71.6
12.7
43.6
EAEFNet
29.3
0.02
60.1
0.0
38.1
TABLE I: Quantitative experiments on the constructed IVMSD dataset
MFNet
Car
Person
Bike
Curve
Car Stop
mIoU
SeAFusion
84.2
71.1
58.7
33.1
20.1
48.8
SegFormer
89.5
73.2
63.8
45.9
20.8
54.7
EGFNet
87.6
69.8
58.8
42.8
33.8
54.8
LASNet
84.2
67.1
56.9
41.1
39.6
54.9
SegMiF
87.8
71.4
63.2
47.5
31.1
56.1
MDRNet+
87.1
69.8
60.9
47.8
34.2
56.8
TABLE II: Quantitative experiments on the MFNet dataset.
PST900
Hand- Drill
BackPack
Frie- Extinguisher
Survivor
mIoU
SeAFusion
65.6
59.6
41.1
29.5
58.9
SegFormer
74.3
86.4
61.1
69.3
78.1
EGFNet
64.7
83.1
71.3
74.3
78.5
LASNet
77.8
86.5
82.8
75.5
84.4
MDRNet+
63.0
76.3
63.5
71.3
74.6
SegMiF
66.0
81.4
76.3
75.5
79.7
TABLE III: Quantitative experiments on the PST900 dataset.
FMB
Car
Truck
T-Lamp
Buil.
mIoU
SeAFusion
76.2
15.1
34.4
80.1
51.9
SegFormer
76.5
38.7
20.9
81.4
56.3
LASNet
73.2
33.1
32.6
80.8
55.7
SegMiF
78.7
42.4
35.6
80.1
58.5
MDRNet+
75.4
27.0
41.4
79.8
55.5
SGFNet
75.0
34.6
45.8
78.2
56.0
TABLE IV: Quantitative experiments on the FMB dataset.
Backbone
FSEC
GMIA
SVCA
mIoU
IVMSD
MFNet
✓
✕
✕
✕
55.8
57.8
✓
✓
✕
✕
57.4
58.5
✓
✓
✓
✕
58.3
58.9
✓
✓
✓
✓
60.2
59.3
TABLE V: Ablation studies of the proposed modules.
IVMSD
Cargo Ship
Fishing Boat
Sand Dredger
Speedboat
mIoU
MRFS [ 34 ]
41.0
29.3
83.8
28.5
56.7
AMDANet [ 45 ]
44.7
28.4
81.2
29.6
56.6
Ours (mit-b4)
56.1
28.8
80.2
26.5
57.9
Ours (ViT-B)
55.8
30.4
81.4
25.7
58.4
Ours (ViT-H)
59.3
28.9
81.7
32.1
60.2
TABLE VI: Ablation study of different backbone on the IVMSD.
Module
Gated
Cargo Ship
Fishing Boat
Sand Dredger
Speedboat
mIoU
FSEC
✕
57.6
29.5
78.4
24.8
57.8
✓
55.8
30.4
84.4
25.7
58.4
GMIA
✕
57.8
26.4
80.9
24.3
57.7
✓
55.8
30.4
84.4
25.7
58.4
SVCA
✕
56.9
27.0
79.3
25.6
57.5
✓
55.8
30.4
84.4
25.7
58.4
TABLE VII: Ablation study of the Gated mechanism in FSEC, GMIA and SVCA module on the IVMSD dataset. The baseline is the DINOv3-Base model.
FSEC
Cargo Ship
Fishing Boat
Sand Dredger
Speedboat
mIoU
w/o Hybrid Attn.
51.7
19.0
79.2
20.7
53.9
w Hybrid Attn.
55.8
30.4
84.4
25.7
58.4
TABLE VIII: Ablation study of the Hybrid Attention in the FSEC module on the IVMSD dataset
Fig. 4: The visualization of the qualitative segmentation result.
Model
Mechanism
mIoU (%)
DINOV3 (ViT-Base)
w Simple Fusion
53.9
w Cross-Attn
54.6
w GMIA
55.1
DINOV3 (ViT-Base)
w Frequency Process
54.6
Ours
56.0
DINOV3 (ViT-Base)
w Self-Attn
54.2
TABLE IX: Ablation study for the proposed module and various standard techniques on the IVMSD.
School of New Media and Communication, Tianjin University · School of Artificial Intelligence, Tianjin University · School of Automation, Southeast University +1
School of Computer Science, Wuhan University · School of Computer Science, The University of Sydney · School of Computer Science and Technology, Harbin Institute of Technology