Bridging the Gap between Labeled and Unlabeled Data via Unified Flow with Feature Memory Bank
Authors: Shanwen Wang, Xin Sun, Danfeng Hong, Junyu Dong, Patrick Le Callet
Organizations: Faculty of Data Science, City University of Macau, 999078, SAR Macao, China · School of Automation, Southeast University, Nanjing, 211189, China · Department of Computer Science and Technology, Ocean University of China, Qingdao, 266100 China · Nantes Universit´e, Ecole Centrale Nantes, CAPACITES SAS, CNRS, LS2N, UMR 6004, Nantes, France
Although semi-supervised semantic segmentation (S4) utilizes abundant unlabeled data to reduce manual labeling burdens, independent training of labeled and unlabeled data causes the former to dominate, which severely degrades pseudo-label quality. To address this challenges, we propose a novel remote sensing (RS) S4 method via unified flow with feature memory bank (UFFM). Specifically, UFFM comprises two key innovations: unified flow (UF) and feature memory bank (FMB). The UF is a new training flow that generates less biased pseudo-labels by combining an external visual foundation model (VFM) with an RS domain teacher, and jointly optimizes labeled and pseudo-labeled data under a unified training objective. The FMB is a novel memory module for S4 that dynamically updates class-specific features during training and reduces the feature discrepancy between labeled and unlabeled data through class-feature alignment. To verify the effectiveness of our model, we conduct extensive experiments on RS datasets. The experimental results show the superiority of our method over SOTA S4 methods. Moreover, the results demonstrate the effectiveness of our contributions in bridging the optimization and feature representation gap between labeled and unlabeled data. Our code is released at https://github.com/wangshanwen001/RS-UFFM.
Figures & tables
Fig. 1: (a), (b) and (c) Comparison between the training data flows of previous methods, AllSpark and ours UFFM. (d) Comparison between the accuracy rate of previous methods, AllSpark and ours UFFM.
Fig. 2: Overall architecture of our UFFM model for RS images. Flow paths are color-coded: blue for labeled data, yellow for unlabeled data, black for UF processing, and dashed for loss functions. As shown by the black path in the top row, pseudo-labels in the UFFM model are initially generated by the VFM as class-agnostic masks and subsequently mapped to semantic classes by the teacher. The light blue-gray region highlights the FMB module, which comprises two stages: feature memory bank updating and loss computation, where loss weights are determined via cosine similarity.
Fig. 3: The overall workflow of our FMB. It mainly consists of two stages: the memory bank update stage (the red lines) and the loss computation stage (the blue lines).
Ratio
Model
IoU
mIoU
Urban
Agr.
Range.
Forest
Water
Barren
1%
CCT [ 54 ]
70.86
70.64
11.03
62.44
28.76
27.66
45.23
CPS [ 55 ]
80.94
70.66
1.16
63.91
27.45
0.79
40.82
LSST [ 56 ]
79.35
73.41
21.60
60.76
30.40
25.64
48.53
FixMatch [ 31 ]
82.51
74.10
18.79
67.65
44.72
32.74
53.42
UniMatch [ 32 ]
80.54
70.72
20.71
65.48
34.09
9.24
46.80
TABLE I: Comparison results with SOTA methods on DeepGlobe dataset. The best results are highlighted in bold. IoU and mIoU are represented as percentages.
Ratio
Model
IoU
mIoU
Building
Low vegetation
Tree
Car
Impervious surfaces
1%
CCT [ 54 ]
54.48
61.28
48.56
52.95
60.71
55.59
CPS [ 55 ]
59.35
69.16
62.89
59.88
66.33
63.52
LSST [ 56 ]
68.74
75.24
54.74
62.09
68.80
65.92
FixMatch [ 31 ]
76.95
71.59
64.71
65.85
72.81
70.38
UniMatch [ 32 ]
76.52
70.99
65.44
66.62
72.64
70.44
TABLE II: Comparison results with SOTA methods on ISPRS-Potsdam dataset. The best results are highlighted in bold. IoU and mIoU are represented as percentages.
Fig. 4: Visual comparison of semantic segmentation results with different semisupervised methods on the ISPRS-Potsdam dataset.
Fig. 5: Visual comparison of semantic segmentation results with different semisupervised methods on the DeepGlobe dataset.
Dataset
Network
Labeled mIoU
Unlabeled mIoU
mIoU
DeepGlobe
Baseline
91.32
56.89
57.35
UF
97.83
59.74
60.94
UF+FMB
98.49
60.97
61.55
ISPRS-Potsdam
Baseline
88.09
70.82
71.51
UF
94.50
74.48
75.74
UF+FMB
96.92
77.01
77.15
TABLE III: Ablation of the Components.
Fig. 6: Hyperparameters Analysis τconsistency for UF.
Fig. 7: The ablation study of the hyperparameters τconf for UF.
Dataset
Backbone
mIoU
Params
DeepGlobe
DINOv2-small
61.55
24.8M
DINOv2-base
67.38
97.5M
DINOv2-large
68.74
335.6M
DINOv3-small
62.81
24.3M
DINOv3-base
67.92
96.6M
DINOv3-large
69.01
334.4M
TABLE IV: The Backbone Investigation.
Dataset
Network
Labeled mIoU
Unlabeled mIoU
mIoU
DeepGlobe
Baseline
91.32
56.89
57.35
VFM
92.26
57.35
58.56
Full UF
97.83
59.74
60.94
ISPRS-Potsdam
Baseline
88.09
70.82
71.51
VFM
89.86
71.67
72.11
Full UF
94.50
74.48
75.74
TABLE V: Ablation Study of UF.
Fig. 8: t-SNE Visualizations of Learned Representations.
National Taiwan University of Science and Technology, Taipei, Taiwan · National Taiwan University, Taipei, Taiwan · Chang Gung University, Taoyuan, Taiwan +3
School of Computer Science and Engineering, Southeast University, Nanjing, China. · School of Electrical Engineering, Southeast University, Nanjing, China. · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, Nanjing, China.