Organizations: College of Computer Science, Sichuan University, Chengdu 610065, China · Department of Computer and Information Sciences, Northumbria University, Newcastle upon Tyne NE1 8ST, UK · Tianfu Jincheng Laboratory, Chengdu 610093, China
Cross-modal retrieval (CMR) aims to bridge the semantic gaps between modalities by learning shared representations. Unfortunately, due to the dynamism and inconsistencies in real-world data acquisition, multimodal datasets frequently encounter severe modality incompleteness. Existing incomplete CMR methods adhere to a label-driven paradigm, relying heavily on categorical annotations as the semantic benchmark. However, label noise is difficult to avoid and often coexists with modality incompleteness. To address this challenge, we propose a novel semantic prior-guided framework for incomplete cross-modal retrieval with noisy labels, termed SPIRN. Specifically, SPIRN uses fixed semantic priors as a consistent reference for robust representation learning, noisy-label handling, and missing-modality completion. Based on this reference, Semantic Priors Guidance (SPG) establishes a stable category-level semantic space, Quality-aware Robust Learning (QRL) dynamically evaluates sample quality to distinguish clean samples from noisy ones and handles noise via reverse learning, Prior-guided Global Label Calibration (PGLC) rectifies corrupted annotations for refined supervision, and Confidence-driven Selective Imputation (CSI) recovers missing modalities from a cross-modal memory bank while filtering unreliable imputations with an increasing threshold. To the best of our knowledge, this is the first framework to jointly tackle the dual challenges of modality incompleteness and label noise in the CMR community. Extensive experiments on four widely used benchmark datasets demonstrate that SPIRN outperforms 11 state-of-the-art methods in various scenarios involving both missing modalities and noisy labels.
Figures & tables
Figure 1: The motivation of SPIRN. Conventional ICMR methods rely heavily on labels and are vulnerable to noisy supervision, which may lead to semantic confusion. In contrast, SPIRN filters and exploits high-quality imputed data, thereby alleviating the impact of noisy labels.
Figure 2: The framework of SPIRN consists of four core components: Semantic Priors Guidance (SPG), Quality-aware Robust Learning (QRL), Prior-guided Global Label Calibration (PGLC), and Confidence-driven Selective Imputation (CSI).
Missing
Method
Wikipedia
XMedia
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
Easy
ALGCN
37.2
34.4
28.6
27.6
19.4
19.7
14.0
14.7
86.3
85.6
76.0
73.2
51.1
46.0
13.5
11.4
UCCH
35.6
34.3
35.6
34.3
35.6
34.3
35.6
34.3
73.6
73.5
73.6
73.5
73.6
73.5
73.6
73.5
GNN4CMR
44.4
40.5
40.7
37.2
31.6
29.1
19.9
19.6
85.1
84.8
79.3
80.3
70.5
71.3
42.9
42.7
RONO
49.4
45.2
48.8
45.1
47.4
43.3
37.3
35.4
90.3
90.7
89.3
89.8
86.3
86.8
81.6
81.0
Table 1: Comparison of mAP (%) on Wikipedia and XMedia under different completeness and noise settings.
Missing
Method
INRIA-Websearch
XMediaNet
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
Easy
ALGCN
32.3
30.3
19.5
16.9
6.4
4.1
2.0
1.7
21.4
26.1
8.0
10.6
1.7
2.2
0.8
1.1
UCCH
35.6
35.5
35.6
35.5
35.6
35.5
35.6
35.5
24.1
25.8
24.1
25.8
24.1
25.8
24.1
25.8
GNN4CMR
45.5
45.2
40.2
38.7
34.3
32.7
28.5
26.8
52.0
52.4
46.6
47.2
41.2
41.8
36.6
37.4
RONO
44.4
44.9
43.0
43.0
40.4
40.1
38.8
37.7
35.1
33.7
37.0
34.8
37.0
35.8
30.3
30.5
Table 2: Comparison of mAP (%) on INRIA-Websearch and XMediaNet under different completeness and noise settings.
Figure 3: Precision-recall curves on XMedia and XMediaNet under the medium missing setting with a noise ratio of 0.6.
XMedia
XMediaNet
Methods
I2T
T2I
Mean
I2T
T2I
Mean
SPIRN-1
83.97
84.27
84.12
15.64
16.65
16.15
SPIRN-2
89.57
89.81
89.69
55.43
55.68
55.56
SPIRN-3
82.65
83.07
82.86
55.18
54.80
54.99
SPIRN
89.68
90.00
89.84
55.79
55.82
55.81
Table 3: The mAP (%) results of the ablation study on the XMedia and XMediaNet datasets.
Clean (%)
XMedia
XMediaNet
Easy
Medium
Hard
Easy
Medium
Hard
80.0
98.6
96.9
94.0
96.9
91.3
80.6
60.0
98.3
96.4
91.0
96.9
91.0
80.9
40.0
97.8
95.9
91.3
96.4
90.3
80.5
20.0
96.8
93.4
83.6
90.7
81.4
68.8
Table 4: The accuracy (%) of global label calibration under different label noise and missing data settings.
Figure 4: The ratio (%) of selected clean samples under different missing settings.
Figure 5: The mAP (%) of three variants with the 0.6 noise ratio under different missing settings.
Symbol
Description
Unit
Value
D
Embedding size
dimension
512
nb
Batch size
samples
128/256
Ewarm
Warm-up endpoint
epoch
10
Enoisy
Robust-stage endpoint
epoch
40
Emax
Total epochs
epoch
100
α
Reverse-loss weight
scalar
1.0
Table 5: Key hyperparameter settings of SPIRN.
Method
Reference
Noisy
Missing
ALGCN
TMM’22
No
No
UCCH
TPAMI’22
No
No
GNN4CMR
TPAMI’22
No
Yes
RONO
CVPR’23
Yes
No
CIRH
TKDE’23
No
No
NCH
TMM’23
No
Yes
Table 6: The methods of 11 state-of-the-art baselines.
Figure 6: Precision-recall curves on four datasets under the medium missing setting with a noise ratio of 0.6.
Noise
Seed
XMedia
XMediaNet
Easy
Medium
Hard
Easy
Medium
Hard
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
0.40
2021
90.27
90.98
90.30
91.08
87.60
88.29
59.16
59.06
55.65
55.58
49.16
48.68
2022
90.73
91.34
90.02
90.96
86.91
87.60
59.69
59.57
55.57
55.98
49.06
48.93
2023
90.87
91.83
89.93
90.71
87.19
87.88
59.70
59.29
55.81
55.91
49.13
48.60
2024
90.79
91.11
90.27
90.88
87.43
88.16
59.23
59.32
55.43
55.79
49.03
48.71
Table 7: The mAP (%) results under different random seeds on XMedia and XMediaNet, together with the mean ( μ ) and standard deviation ( σ ).
Dataset
Method
Noise Ratio
0.2
0.4
0.6
0.8
I2T
T2I
Mean
I2T
T2I
Mean
I2T
T2I
Mean
I2T
T2I
Mean
XMedia
ALGCN
47.1
15.0
31.0
29.5
11.9
20.7
13.1
8.5
10.8
8.5
7.2
7.8
UCCH
55.5
59.9
57.7
55.5
59.9
57.7
55.5
59.9
57.7
55.5
59.9
57.7
GNN4CMR
57.7
61.9
59.8
48.1
52.4
50.2
36.3
38.3
37.3
28.5
28.5
28.5
RONO
53.6
52.0
52.8
48.7
46.7
47.7
40.0
38.4
39.2
35.3
33.8
34.5
Table 8: Comparison of mAP (%) on XMedia and XMediaNet under a 90% missing-modality setting and four noise ratios.
Figure 7: The mAP (%) results under the medium missing scene for different α , β , and η .
Dataset
Method
Noise Ratio
0.1
0.2
0.3
0.4
I2T
T2I
Mean
I2T
T2I
Mean
I2T
T2I
Mean
I2T
T2I
Mean
XMedia
ALGCN
85.6
84.4
85.0
82.4
79.8
81.1
75.3
70.3
72.8
65.0
56.9
61.0
UCCH
72.7
74.4
73.5
72.7
74.4
73.5
72.7
74.4
73.5
72.7
74.4
73.5
GNN4CMR
84.7
85.3
85.0
82.9
81.9
82.4
78.7
78.8
78.7
76.1
76.5
76.3
RONO
89.1
90.0
89.6
89.3
90.1
89.7
86.9
88.1
87.5
86.4
86.5
86.4
Table 9: Comparison of mAP (%) on XMedia and XMediaNet under asymmetric label noise with varying noise ratios in the medium missing setting.
Figure 8: The mAP (%) results of SPIRN under different epoch combinations.
Figure 9: The mAP curves of SPIRN under the medium missing modality setting with a label noise ratio of 0.6.
Figure 10: Training time (s) and GPU memory consumption (MB) of different methods under varying modality-completeness settings, averaged over four label-noise ratios.
Dataset
Method
Noise Ratio
0.2
0.4
0.6
0.8
I2T
T2I
Mean
I2T
T2I
Mean
I2T
T2I
Mean
I2T
T2I
Mean
XMediaNet
SPIRN-M
33.4
36.5
35.0
30.8
33.4
32.1
24.4
26.5
25.5
15.6
18.0
16.8
SPIRN-E
31.8
34.1
32.9
27.3
29.4
28.4
20.5
22.0
21.2
18.3
20.6
19.4
SPIRN-L
40.9
43.8
42.3
40.0
42.9
41.5
37.2
39.9
38.5
26.5
27.9
27.2
SPIRN-R
55.4
55.4
55.4
55.2
55.2
55.2
55.5
54.6
55.0
52.9
51.8
52.3
Table 10: Comparison of different semantic-prior constructions on XMediaNet under the medium missing modality setting and four noise ratios.
Figure 11: t-SNE visualizations learned from the training data on the XMedia dataset under the medium missing scene with a 0.6 label noise ratio, where different colors denote different semantic categories.
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China · Zhongguancun Academy, Beijing 100094, China · Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China +1