Composed Image Retrieval (CIR) is a challenging image retrieval paradigm. It aims to retrieve target images from large-scale image databases that are consistent with the modification semantics, based on a multimodal query composed of a reference image and modification text. Although existing methods have made significant progress in cross-modal alignment and feature fusion, a key flaw remains: the neglect of contextual information in discriminating matching samples. However, addressing this limitation is not an easy task due to two challenges: 1) implicit dependencies and 2) the lack of a differential amplification mechanism. To address these challenges, we propose a dual-patH composItional coNtextualized neTwork (HINT), which can perform contextualized encoding and amplify the similarity differences between matching and non-matching samples, thus improving the upper performance of CIR models in complex scenarios. Our HINT model achieves optimal performance on all metrics across two CIR benchmark datasets, demonstrating the superiority of our HINT model. Codes are available at https://github.com/zh-mingyu/HINT.
Figures & tables
Figure 1: An example of the CIR task and HINT’s main idea.
Figure 2: HINT framework: (a) Dual Context Extraction, (b) Quantification of Contextual Relevance, (c) Dual-Path Consistency Constraints.
Method
FashionIQ
CIRR
Dresses
Shirts
Tops&Tees
Avg
R@ k
R subset @ k
Avg
R@ 10
R@ 50
R@ 10
R@ 50
R@ 10
R@ 50
R@ 10
R@ 50
k = 1
k = 5
k = 10
k = 50
k = 1
k = 2
k = 3
TG-CIR [ 25 ] (ACM MM’23)
45.22
69.66
52.60
72.52
56.14
77.10
51.32
73.09
45.25
78.29
87.16
97.30
72.84
89.25
95.13
75.57
SSN [ 28 ] (AAAI’24)
34.36
60.78
38.13
61.83
44.26
69.05
38.92
63.89
43.91
77.25
86.48
97.45
71.76
88.63
95.54
74.51
MMF [ 20 ] (ICASSP’24)
40.45
65.29
44.85
66.29
49.26
70.98
44.85
67.52
43.80
77.37
86.94
97.71
71.39
88.16
95.15
74.38
CALA [ 12 ] (SIGIR’24)
42.38
66.08
46.76
68.16
50.93
73.42
46.69
69.22
49.11
81.21
89.59
98.00
76.27
91.04
96.46
78.74
Table 1: Performance comparison on FashionIQ and CIRR relative to R@ k (%). The overall best results are in bold, while the second-best results are underlined. The Avg metric in CIRR denotes (R@ 5 + R subset @ 1 ) / 2.
Method
FashionIQ
CIRR
R@ 10
Δ
R@ 50
Δ
Avg
Δ
Dual Context Extraction (DCE)
w/o VCM
56.86
-1.62
77.03
-0.90
80.16
-1.41
w/o CCM
56.66
-1.82
77.07
-0.86
80.21
-1.36
w/o DCE
55.35
-3.13
75.91
-2.02
79.96
-1.61
Quantification of contextual relevance (QCR)
Table 2: Ablation study on FashionIQ and CIRR datasets. We compute Avg-R@ 10 , Avg-R@ 50 for FashionIQ, and Avg (mean of R@ 5 and R subset @ 1 ) for CIRR, respectively.
Composed Image Retrieval (CIR) uses a reference image and a modification text as a query to retrieve a target image satisfying the requirement of modifying the reference image according to the text instructions''. However, existing CIR methods face two limitations: (1) frequency bias leading to Rare Sample Neglect'', and (2) susceptibility of similarity scores to interference from hard negative samples and noise. To address these limitations, we confront two key challenges: asymmetric rare semantic localization and robust similarity estimation under hard negative samples. To solve these challenges, we propose the Modification frEquentation-rarity baLance neTwork MELT. MELT assigns increased attention to rare modification semantics in multimodal contexts while applying diffusion-based denoising to hard negative samples with high similarity scores, enhancing multimodal fusion and matching. Extensive experiments on two CIR benchmarks validate the superior performance of MELT. Codes are available at https://github.com/luckylittlezhi/MELT.
Guozhi Qiu, Zhiwei Chen, Zixu Li +4
School of Software, Shandong University · Southern University of Science and Technology
Composed Image Retrieval (CIR) is a multimodal retrieval task where a query consists of a reference image and a textual modification, and the goal is to retrieve a target image satisfying both. In principle, strong performance on CIR benchmarks is assumed to require multimodal composition, i.e., combining complementary information from reference image and textual modification. In this work, we show that this assumption does not always hold. Across four widely used CIR benchmarks and eleven Generalist Multimodal Embedding models, a large fraction of queries can be solved using a single modality (from 32.2% to 83.6%), revealing pervasive unimodal shortcuts. Thus, high CIR performance can arise from unimodal signals rather than true multimodal composition. To better understand this issue, we perform a two-stage audit. First, we identify shortcut-solvable queries through cross-model analysis. Second, we conduct human validation on 4,741 shortcut-free queries, of which only 1,689 are well-formed, with common issues including ambiguous edits and mismatched targets. Re-evaluating models on this validated subset reveals qualitatively different behaviour: queries can no longer be solved with a single modality, and successful retrieval requires combining both inputs. While accuracy decreases, reliance on multimodal information increases. Overall, current CIR benchmarks conflate shortcut-solvable, noisy, and genuinely compositional queries, leading to an overestimation of model capability in multimodal composition.
Matteo Attimonelli, Alessandro De Bellis, Aryo Pradipta Gema +8
Composed Image Retrieval (CIR) is a flexible image retrieval paradigm that enables users to accurately locate the target image through a multimodal query composed of a reference image and modification text. Although this task has demonstrated promising applications in personalized search and recommendation systems, it encounters a severe challenge in practical scenarios known as the Noise Triplet Correspondence (NTC) problem. This issue primarily arises from the high cost and subjectivity involved in annotating triplet data. To address this problem, we identify two central challenges: the precise estimation of composed semantic discrepancy and the insufficient progressive adaptation to modification discrepancy. To tackle these challenges, we propose a cHrono-synergiA roBust progressIve learning framework for composed image reTrieval (HABIT), which consists of two core modules. First, the Mutual Knowledge Estimation Module quantifies sample cleanliness by calculating the Transition Rate of mutual information between the composed feature and the target image, thereby effectively identifying clean samples that align with the intended modification semantics. Second, the Dual-consistency Progressive Learning Module introduces a collaborative mechanism between the historical and current models, simulating human habit formation to retain good habits and calibrate bad habits, ultimately enabling robust learning under the presence of NTC. Extensive experiments conducted on two standard CIR datasets demonstrate that HABIT significantly outperforms most methods under various noise ratios, exhibiting superior robustness and retrieval performance. Codes are available at https://github.com/Lee-zixu/HABIT