Composed Image Retrieval (CIR) aims to retrieve target images based on a reference image and modified texts. However, existing methods often struggle to extract the correct semantic cues from the reference image that best reflect the user's intent under textual modification prompts, resulting in interference from irrelevant visual noise. In this paper, we propose a novel Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning (MCoT-MVS) for CIR, integrating attention-aware multi-level vision features guided by reasoning cues from a multi-modal large language model (MLLM). Specifically, we leverage an MLLM to perform chain-of-thought reasoning on the multimodal composed input, generating the retained, removed, and target-inferred texts. These textual cues subsequently guide two reference visual attention selection modules to selectively extract discriminative patch-level and instance-level semantics from the reference image. Finally, to effectively fuse these multi-granular visual cues with the modified text and the imagined target description, we design a weighted hierarchical combination module to align the composed query with target images in a unified embedding space. Extensive experiments on two CIR benchmarks, namely CIRR and FashionIQ, demonstrate that our approach consistently outperforms existing methods and achieves new state-of-the-art performance. Code and trained models are publicly released.
Figures & tables
Figure 1. Compared with the traditional CIR model (B) and cross-modal attention-based CIR model (C), in this paper, we introduce multimodal chain-of-thought reasoning to explicitly extract the relevant and irrelevant contents for reference visual information selection (A). In this way, while effectively focusing on fine-grained useful references, we can significantly avoid the attention noise interference driven by visual-linguistic correlation as in method (C).
Figure 2. Illustration of our proposed MCoT-MVS. It primarily consists of the Multimodal Chain-of-Thought (CoT) Reasoning module, Patch-level Visual Reference Selection (PVRS), the Instance-level Visual Reference Selection module (IVRS), and a Weighted Hierarchical Combination (WHC). The multimodal CoT reasoning module first reasons the composed query into retained and deleted contents, as well as the potential target context, based on a pre-trained MLLM. Then, the useful patch-level and instance-level reference visual representations are semantically selected by the explicitly reasoned retained and deleted texts to retain the correct user intent and to remove visual noise. Finally, the WHC fuses the selected representations with multiple target texts into an attention-aware query, aligning with the target image.
Method
R@1
R@5
R@10
R@50
Avg
Traditional Model-Based Methods
TIRG ( Vo et al., 2019 ) (CVPR′19)
14.61
48.37
64.08
90.03
54.27
CIRPLANT ( Liu et al., 2021 ) (ICCV′19)
19.55
52.55
68.39
92.38
58.22
ARTEMIS ( Delmas et al., 2022 ) (ICLR′22)
16.96
46.10
61.31
87.73
53.03
VLP Model-Based Methods
CLIP4CIR ( Baldrati et al., 2023 ) (TOMM′23)
38.53
69.98
81.86
95.93
71.58
Table 1. Performance comparison on CIRR from the open-world scene. The best and the second-best scores are marked in bold and underlined fonts, respectively.
Dresses
Shirts
Tops&Tees
Avg
Method
R@10
R@50
R@10
R@50
R@10
R@50
R@10
R@50
Traditional Model-Based Methods
TIRG ( Vo et al., 2019 ) (CVPR′19)
14.87
34.66
18.26
37.89
19.08
39.62
17.40
37.39
CIRPLANT ( Liu et al., 2021 ) (ICCV′19)
17.45
40.41
17.53
38.81
21.64
45.38
18.87
41.53
ARTEMIS ( Delmas et al., 2022 ) (ICLR′22)
27.16
52.40
21.78
43.64
29.20
54.83
26.05
50.29
VLP Model-Based Methods
Table 2. Performance comparison on FashionIQ from fashion products. The best and the second-best scores are marked in bold and underlined fonts, respectively.
NO.
Variant
Performance
WHC
Sum
Combiner
M-T
T-T
PVRS
IVRS
Recall@1
Recall@5
Recall@10
Recall@50
Mean
1
√
-
-
√
√
√
√
55.33
84.75
91.45
98.55
82.52
2
-
√
-
√
√
√
√
52.02
83.01
90.80
98.36
81.05
3
-
-
√
-
√
√
√
45.00
76.48
85.35
96.87
75.93
4
-
-
√
√
-
√
√
54.50
83.69
91.28
98.56
81.98
5
-
√
√
-
√
-
53.49
84.07
91.21
98.55
81.83
Table 3. Ablation study for MCoT-MVS on CIRR. “M-T" and “T-T" indicate the modification texts and the reasoned target texts.
Figure 3. Visualization of the learned attention weights from PVRS and IVRS modules guided by the inferred explicit retain or delete modification intents from the MLLM-based multimodal CoT reasoning, as well as the weight distributions of weighted hierarchical combination modules (best viewed in color).
Model
R@1
R@5
R@10
R@50
w/o Multi-step MCot
54.29
84.45
91.15
98.60
w/ Multi-step MCot
55.33
84.75
91.45
98.55
Table 4. Retrieval performance (%) of MCoT-MVS with different reasoning patterns.
Composed image retrieval (CIR) aims to retrieve a target image that depicts a reference image modified by a textual description. While recent vision-language models (VLMs) achieve promising CIR performance by embedding images and text into a shared space for retrieval, they often fail to reason about what to preserve and what to change. This limitation hinders interpretability and yields suboptimal results, particularly in fine-grained domains like fashion. In this paper, we introduce FIRE-CIR, a model that brings compositional reasoning and interpretability to fashion CIR. Instead of relying solely on embedding similarity, FIRE-CIR performs question-driven visual reasoning: it automatically generates attribute-focused visual questions derived from the modification text, and verifies the corresponding visual evidence in both reference and candidate images. To train such a reasoning system, we automatically construct a large-scale fashion-specific visual question answering dataset, containing questions requiring either single- or dual-image analysis. During retrieval, our model leverages this explicit reasoning to re-rank candidate results, filtering out images inconsistent with the intended modifications. Experimental results on the Fashion IQ benchmark show that FIRE-CIR outperforms state-of-the-art methods in retrieval accuracy. It also provides interpretable, attribute-level insights into retrieval decisions.
François Gardères, Camille-Sovanneary Gauthier, Jean Ponce +1
Louis Vuitton · Inria, École normale supérieure, CNRS, PSL Research University · Courant Institute of Mathematical Sciences and Center for Data Science, New York University
Composed Image Retrieval (CIR) uses a reference image and a modification text as a query to retrieve a target image satisfying the requirement of modifying the reference image according to the text instructions''. However, existing CIR methods face two limitations: (1) frequency bias leading to Rare Sample Neglect'', and (2) susceptibility of similarity scores to interference from hard negative samples and noise. To address these limitations, we confront two key challenges: asymmetric rare semantic localization and robust similarity estimation under hard negative samples. To solve these challenges, we propose the Modification frEquentation-rarity baLance neTwork MELT. MELT assigns increased attention to rare modification semantics in multimodal contexts while applying diffusion-based denoising to hard negative samples with high similarity scores, enhancing multimodal fusion and matching. Extensive experiments on two CIR benchmarks validate the superior performance of MELT. Codes are available at https://github.com/luckylittlezhi/MELT.
Guozhi Qiu, Zhiwei Chen, Zixu Li +4
School of Software, Shandong University · Southern University of Science and Technology
Composed image retrieval (CIR) aims to retrieve a desired target image from a query consisting of a reference image and a modification text. This task exhibits an unusual representational asymmetry: the modification text specifies a transition from the reference state, whereas retrieval candidates depict completed target states. This creates a representation mismatch for zero-shot methods that query pretrained vision-language spaces directly with transformation-oriented language. We study this mismatch and reformulate zero-shot composed image retrieval as target-state reconstruction followed by retrieval. We instantiate this formulation with ASAP-CIR, a training-free framework that reconstructs a static target representation using a frozen multimodal large language model (MLLM). The representation combines multiple holistic descriptions with a variable set of importance-weighted atomic semantics, thereby preserving both overall target identity and fine-grained visual constraints. Retrieval then integrates holistic state alignment, atomic constraint grounding, and calibrated target-state evidence aggregation. A controlled text-only diagnostic shows that target-side static query formulations achieve more reliable retrieval than dynamic composed query formulations, particularly when source-state semantics must be suppressed or transformed. Experiments on FashionIQ, CIRR, and CIRCO further characterize the effectiveness and limitations of this representation principle, with the clearest gains on the multi-target CIRCO benchmark. These results show that how composed intent is represented before retrieval is a consequential design choice, distinct from the choice of retrieval backbone itself.
Yihe Zhao, Songhe Feng
School of Computer Science and Technology, Beijing Jiaotong University, Beijing, 100044, China.