Organizations: College of Computer Science, Sichuan University, Chengdu 610065, China · Department of Computer and Information Sciences, Northumbria University, Newcastle upon Tyne NE1 8ST, UK · Tianfu Jincheng Laboratory, Chengdu 610093, China
Cross-modal retrieval (CMR) aims to bridge the semantic gaps between modalities by learning shared representations. Unfortunately, due to the dynamism and inconsistencies in real-world data acquisition, multimodal datasets frequently encounter severe modality incompleteness. Existing incomplete CMR methods adhere to a label-driven paradigm, relying heavily on categorical annotations as the semantic benchmark. However, label noise is difficult to avoid and often coexists with modality incompleteness. To address this challenge, we propose a novel semantic prior-guided framework for incomplete cross-modal retrieval with noisy labels, termed SPIRN. Specifically, SPIRN uses fixed semantic priors as a consistent reference for robust representation learning, noisy-label handling, and missing-modality completion. Based on this reference, Semantic Priors Guidance (SPG) establishes a stable category-level semantic space, Quality-aware Robust Learning (QRL) dynamically evaluates sample quality to distinguish clean samples from noisy ones and handles noise via reverse learning, Prior-guided Global Label Calibration (PGLC) rectifies corrupted annotations for refined supervision, and Confidence-driven Selective Imputation (CSI) recovers missing modalities from a cross-modal memory bank while filtering unreliable imputations with an increasing threshold. To the best of our knowledge, this is the first framework to jointly tackle the dual challenges of modality incompleteness and label noise in the CMR community. Extensive experiments on four widely used benchmark datasets demonstrate that SPIRN outperforms 11 state-of-the-art methods in various scenarios involving both missing modalities and noisy labels.
Figures & tables
Figure 1: The motivation of SPIRN. Conventional ICMR methods rely heavily on labels and are vulnerable to noisy supervision, which may lead to semantic confusion. In contrast, SPIRN filters and exploits high-quality imputed data, thereby alleviating the impact of noisy labels.
Figure 2: The framework of SPIRN consists of four core components: Semantic Priors Guidance (SPG), Quality-aware Robust Learning (QRL), Prior-guided Global Label Calibration (PGLC), and Confidence-driven Selective Imputation (CSI).
Missing
Method
Wikipedia
XMedia
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
Easy
ALGCN
37.2
34.4
28.6
27.6
19.4
19.7
14.0
14.7
86.3
85.6
76.0
73.2
51.1
46.0
13.5
11.4
UCCH
35.6
34.3
35.6
34.3
35.6
34.3
35.6
34.3
73.6
73.5
73.6
73.5
73.6
73.5
73.6
73.5
GNN4CMR
44.4
40.5
40.7
37.2
31.6
29.1
19.9
19.6
85.1
84.8
79.3
80.3
70.5
71.3
42.9
42.7
RONO
49.4
45.2
48.8
45.1
47.4
43.3
37.3
35.4
90.3
90.7
89.3
89.8
86.3
86.8
81.6
81.0
Table 1: Comparison of mAP (%) on Wikipedia and XMedia under different completeness and noise settings.
Missing
Method
INRIA-Websearch
XMediaNet
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
Easy
ALGCN
32.3
30.3
19.5
16.9
6.4
4.1
2.0
1.7
21.4
26.1
8.0
10.6
1.7
2.2
0.8
1.1
UCCH
35.6
35.5
35.6
35.5
35.6
35.5
35.6
35.5
24.1
25.8
24.1
25.8
24.1
25.8
24.1
25.8
GNN4CMR
45.5
45.2
40.2
38.7
34.3
32.7
28.5
26.8
52.0
52.4
46.6
47.2
41.2
41.8
36.6
37.4
RONO
44.4
44.9
43.0
43.0
40.4
40.1
38.8
37.7
35.1
33.7
37.0
34.8
37.0
35.8
30.3
30.5
Table 2: Comparison of mAP (%) on INRIA-Websearch and XMediaNet under different completeness and noise settings.
Figure 3: Precision-recall curves on XMedia and XMediaNet under the medium missing setting with a noise ratio of 0.6.
XMedia
XMediaNet
Methods
I2T
T2I
Mean
I2T
T2I
Mean
SPIRN-1
83.97
84.27
84.12
15.64
16.65
16.15
SPIRN-2
89.57
89.81
89.69
55.43
55.68
55.56
SPIRN-3
82.65
83.07
82.86
55.18
54.80
54.99
SPIRN
89.68
90.00
89.84
55.79
55.82
55.81
Table 3: The mAP (%) results of the ablation study on the XMedia and XMediaNet datasets.
Clean (%)
XMedia
XMediaNet
Easy
Medium
Hard
Easy
Medium
Hard
80.0
98.6
96.9
94.0
96.9
91.3
80.6
60.0
98.3
96.4
91.0
96.9
91.0
80.9
40.0
97.8
95.9
91.3
96.4
90.3
80.5
20.0
96.8
93.4
83.6
90.7
81.4
68.8
Table 4: The accuracy (%) of global label calibration under different label noise and missing data settings.
Figure 4: The ratio (%) of selected clean samples under different missing settings.
Figure 5: The mAP (%) of three variants with the 0.6 noise ratio under different missing settings.
Symbol
Description
Unit
Value
D
Embedding size
dimension
512
nb
Batch size
samples
128/256
Ewarm
Warm-up endpoint
epoch
10
Enoisy
Robust-stage endpoint
epoch
40
Emax
Total epochs
epoch
100
α
Reverse-loss weight
scalar
1.0
Table 5: Key hyperparameter settings of SPIRN.
Method
Reference
Noisy
Missing
ALGCN
TMM’22
No
No
UCCH
TPAMI’22
No
No
GNN4CMR
TPAMI’22
No
Yes
RONO
CVPR’23
Yes
No
CIRH
TKDE’23
No
No
NCH
TMM’23
No
Yes
Table 6: The methods of 11 state-of-the-art baselines.
Figure 6: Precision-recall curves on four datasets under the medium missing setting with a noise ratio of 0.6.
Noise
Seed
XMedia
XMediaNet
Easy
Medium
Hard
Easy
Medium
Hard
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
0.40
2021
90.27
90.98
90.30
91.08
87.60
88.29
59.16
59.06
55.65
55.58
49.16
48.68
2022
90.73
91.34
90.02
90.96
86.91
87.60
59.69
59.57
55.57
55.98
49.06
48.93
2023
90.87
91.83
89.93
90.71
87.19
87.88
59.70
59.29
55.81
55.91
49.13
48.60
2024
90.79
91.11
90.27
90.88
87.43
88.16
59.23
59.32
55.43
55.79
49.03
48.71
Table 7: The mAP (%) results under different random seeds on XMedia and XMediaNet, together with the mean ( μ ) and standard deviation ( σ ).
Dataset
Method
Noise Ratio
0.2
0.4
0.6
0.8
I2T
T2I
Mean
I2T
T2I
Mean
I2T
T2I
Mean
I2T
T2I
Mean
XMedia
ALGCN
47.1
15.0
31.0
29.5
11.9
20.7
13.1
8.5
10.8
8.5
7.2
7.8
UCCH
55.5
59.9
57.7
55.5
59.9
57.7
55.5
59.9
57.7
55.5
59.9
57.7
GNN4CMR
57.7
61.9
59.8
48.1
52.4
50.2
36.3
38.3
37.3
28.5
28.5
28.5
RONO
53.6
52.0
52.8
48.7
46.7
47.7
40.0
38.4
39.2
35.3
33.8
34.5
Table 8: Comparison of mAP (%) on XMedia and XMediaNet under a 90% missing-modality setting and four noise ratios.
Figure 7: The mAP (%) results under the medium missing scene for different α , β , and η .
Dataset
Method
Noise Ratio
0.1
0.2
0.3
0.4
I2T
T2I
Mean
I2T
T2I
Mean
I2T
T2I
Mean
I2T
T2I
Mean
XMedia
ALGCN
85.6
84.4
85.0
82.4
79.8
81.1
75.3
70.3
72.8
65.0
56.9
61.0
UCCH
72.7
74.4
73.5
72.7
74.4
73.5
72.7
74.4
73.5
72.7
74.4
73.5
GNN4CMR
84.7
85.3
85.0
82.9
81.9
82.4
78.7
78.8
78.7
76.1
76.5
76.3
RONO
89.1
90.0
89.6
89.3
90.1
89.7
86.9
88.1
87.5
86.4
86.5
86.4
Table 9: Comparison of mAP (%) on XMedia and XMediaNet under asymmetric label noise with varying noise ratios in the medium missing setting.
Figure 8: The mAP (%) results of SPIRN under different epoch combinations.
Figure 9: The mAP curves of SPIRN under the medium missing modality setting with a label noise ratio of 0.6.
Figure 10: Training time (s) and GPU memory consumption (MB) of different methods under varying modality-completeness settings, averaged over four label-noise ratios.
Dataset
Method
Noise Ratio
0.2
0.4
0.6
0.8
I2T
T2I
Mean
I2T
T2I
Mean
I2T
T2I
Mean
I2T
T2I
Mean
XMediaNet
SPIRN-M
33.4
36.5
35.0
30.8
33.4
32.1
24.4
26.5
25.5
15.6
18.0
16.8
SPIRN-E
31.8
34.1
32.9
27.3
29.4
28.4
20.5
22.0
21.2
18.3
20.6
19.4
SPIRN-L
40.9
43.8
42.3
40.0
42.9
41.5
37.2
39.9
38.5
26.5
27.9
27.2
SPIRN-R
55.4
55.4
55.4
55.2
55.2
55.2
55.5
54.6
55.0
52.9
51.8
52.3
Table 10: Comparison of different semantic-prior constructions on XMediaNet under the medium missing modality setting and four noise ratios.
Figure 11: t-SNE visualizations learned from the training data on the XMedia dataset under the medium missing scene with a 0.6 label noise ratio, where different colors denote different semantic categories.
Large-scale web-harvested datasets have fueled the progress of cross-modal retrieval but inevitably suffer from noisy correspondence, which severely degrades model generalization. Existing methods primarily address this by filtering out noise or seeking a substitute label, yet they predominantly remain bound by a "Discrete Selection" paradigm. We argue that relying on a single discrete proxy induces Single-Point Fragility and Discretization Error. To overcome these limitations, we propose a novel framework, Intra-modal Neighbor-aware Noise Rectification (IN2R), which shifts the paradigm from searching for a substitute to synthesizing a reliable supervision target. Leveraging the intrinsic geometric stability of intra-modal data, IN2R employs a Graph Refiner to perform relational reasoning over neighbors retrieved from a dynamic Cross-Model Memory. Instead of propagating discrete labels, our method synthesizes a continuous, soft prototype that reflects the consensus of the local semantic neighborhood, effectively rectifying inter-modal misalignment. Extensive experiments on Flickr30K, MS-COCO, and CC152K demonstrate that IN2R significantly outperforms state-of-the-art methods. Our code and pre-trained models are publicly available at https://github.com/liuyyy111/IN2R.
Yang Liu, Wentao Feng, Shu-Dong Huang +2
College of Computer Science, Sichuan University, China · School of Computer Sci
Multimodal learning robust to missing modalities is essential for real-world applications. Existing methods mainly focus on inter-modality missing, where entire modalities are absent, while overlooking intra-modality degradation, where modalities are present but severely corrupted. In practice, these two types of missing often coexist, making existing approaches ineffective. To address this limitation, we propose General Incomplete Multimodal Learning (GIML), a unified framework that simultaneously handles both inter-modality missing and intra-modality degradation through dynamic quality perception. Specifically, GIML models heterogeneous missing patterns as continuous modality information degradation, enabling degradation-aware adaptive fusion. To achieve reliable quality perception, we introduce a Noise-aware Quality Estimator that learns the mapping from corrupted features to noise intensity through controlled noise injection. Furthermore, we propose a Noise-Semantic Decoupled module that separates semantic information from noise interference. This improves robustness and generalization to unseen corruption patterns. Extensive experiments across datasets with diverse modality types demonstrate the effectiveness and generality of GIML. Code is available at: https://github.com/Yu-Five/GIML.
Xiangyu Meng, Shicai Wei
University of Electronic Science and Technology of China
Most existing multimodal sentiment analysis approaches assume access to complete multimodal inputs. However, real-world applications frequently encounter incomplete or corrupted modalities, posing a critical challenge. Although several methods have been proposed to tackle this issue, they mainly rely on data imputation and heuristic coordination constraints, which fail to effectively extract and leverage task-relevant information from the incomplete multimodal data. To address this challenge, we propose a unified framework termed Mutual Information Disentanglement with uncertainty-Aware fuSion (MIDAS), which effectively restructures multimodal representations under incomplete conditions. MIDAS adopts a variational modeling strategy to represent each modality with multivariate Gaussian latent variables and further decomposes them into shared and exclusive factors. To obtain reliable representations, we design a minimax objective that minimizes the mutual information between shared and exclusive spaces for stable disentanglement, while maximizing the mutual information among shared spaces across modalities to enhance semantic alignment. In addition, an uncertainty-aware fusion mechanism is introduced, where posterior variance is leveraged as a reliability indicator to adaptively weight latent features during fusion, ensuring robust integration even when modalities are incomplete. Extensive experiments on three widely used datasets show that MIDAS achieves strong and consistent performance gains over competitive baselines across a wide range of incomplete settings, demonstrating its effectiveness and robustness for incomplete data scenarios.
Yuhua Wen, Yingying Zhou, Qifei Li +4
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China · Zhongguancun Academy, Beijing 100094, China · Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China +1