FIRE-CIR: Fine-grained Reasoning for Composed Fashion Image Retrieval
Authors: François Gardères, Camille-Sovanneary Gauthier, Jean Ponce, Shizhe Chen
Organizations: Louis Vuitton · Inria, École normale supérieure, CNRS, PSL Research University · Courant Institute of Mathematical Sciences and Center for Data Science, New York University
Composed image retrieval (CIR) aims to retrieve a target image that depicts a reference image modified by a textual description. While recent vision-language models (VLMs) achieve promising CIR performance by embedding images and text into a shared space for retrieval, they often fail to reason about what to preserve and what to change. This limitation hinders interpretability and yields suboptimal results, particularly in fine-grained domains like fashion. In this paper, we introduce FIRE-CIR, a model that brings compositional reasoning and interpretability to fashion CIR. Instead of relying solely on embedding similarity, FIRE-CIR performs question-driven visual reasoning: it automatically generates attribute-focused visual questions derived from the modification text, and verifies the corresponding visual evidence in both reference and candidate images. To train such a reasoning system, we automatically construct a large-scale fashion-specific visual question answering dataset, containing questions requiring either single- or dual-image analysis. During retrieval, our model leverages this explicit reasoning to re-rank candidate results, filtering out images inconsistent with the intended modifications. Experimental results on the Fashion IQ benchmark show that FIRE-CIR outperforms state-of-the-art methods in retrieval accuracy. It also provides interpretable, attribute-level insights into retrieval decisions.
Figures & tables
Figure 1: Given a CIR query, our model analyzes the presence of the requested changes in a list of candidate images, to re-rank them depending on their relevance with respect to the modification text. The target image is framed in green.
Figure 2: Overview of the FIRE-CIR model. Left: VQA score computation. The modification text is decomposed into a set of visual questions about the candidate image, which are answered using a VQA model. The predicted answers are then used to compute the VQA score, measuring the relevance of the candidate image with respect to the text. The set of questions and their answers give an interpretable insight in the relevance evaluation of each image. Right: CIR inference. Given the CIR query and a set of candidate images, FIRE-CIR computes the VQA score of each candidate image, and combines it with the score returned by another CIR method to refine the ranking of the retrieved images.
Evaluation type
Statistics
Value
Automatic
Average number of questions per CIR triplet
3.7
Dual-image inputs
38 %
Manual
One characteristic per question
97 %
No question repetition
84 %
Hallucinations
6 %
Correct expected answers
100 %
Table 1: Statistics on the question generation process, on the Fashion IQ validation dataset. The manual evaluation is performed over 136 questions randomly sampled from the three clothing types.
Model version
Automatic eval.
Manual eval.
Pre-trained
67.10 %
75.83 %
Fine-tuned
81.92 %
83.67 %
Table 2: Evaluation of our VQA model, in its pre-trained and fine-tuned version. The automatic evaluation is performed over the Fashion IQ validation target images, while the manual evaluation is performed over a sample of 600 Fashion IQ validation images.
Model
Dresses
Shirts
Tops&tees
Average
R@10
R@50
R@10
R@50
R@10
R@50
R@10
R@50
Global
SQUARE [ 24 ]
45.04
62.51
37.68
60.19
49.87
69.25
44.20
63.98
54.09
ReRanking [ 16 ]
48.14
71.34
50.15
71.25
55.23
76.80
51.17
73.13
62.15
DetailFusion [ 26 ]
51.34
74.05
58.12
75.95
61.22
80.09
56.89
76.70
66.79
CLIP4CIR [ 2 ]
39.17
64.20
44.80
65.36
47.58
71.09
43.85
66.88
55.37
+ VQA4CIR [ 3 ]
40.91
65.13
45.62
65.68
49.21
71.22
45.24
67.34
56.29
Table 3: Results on the Fashion IQ validation split for state-of-the-art CIR methods. Best results are highlighted in bold.
Model
FIRE-CIR activated
Fashion IQ
enhFashionIQ
Refined-FashionIQ
Recall
MRR
Recall
MRR
Recall
MRR
CLIP4CIR [ 2 ]
✗
55.37
0.2403
66.85
0.2737
73.89
0.4028
✓
58.31
0.2746
73.87
0.3604
78.09
0.4651
SPRC [ 1 ]
✗
64.88
0.3343
78.33
0.4515
82.20
0.5246
✓
65.73
0.3454
80.02
0.4779
83.24
0.5459
FashionBLIP-2 [ 5 ]
✗
65.92
0.3325
87.10
0.5251
82.02
0.5050
Table 4: CIR performance improvement when combining a CIR method with our FIRE-CIR model. For each initial CIR model, the best results are highlighted in bold.
Figure 3: Accuracy of the fine-tuned VQA model depending on the percentage of our automatically-annotated dataset used. For each model version, we measure the accuracy over all the target images in our three different validation sets.
Multiple questions
VQA model fine-tuning
With score merging method
Fashion IQ
Recall@10
Recall@50
Avg. recall
MRR
✗
✗
✗
56.44
76.78
66.61
0.3513
✓
✗
✗
57.28
77.01
67.14
0.3520
✓
✓
✗
57.43
77.30
67.36
0.3584
✓
✓
✓
58.16
77.35
67.75
0.3658
Table 5: Ablation study of the FIRE-CIR model. The best and second-best results are highlighted in bold and underlined, respectively.
Model
Precision
Recall
AUC-PR
AUC-ROC
LLaVA-13B
75.7
88.8
84.4
78.1
InternVL3-1B
80.0
84.2
85.3
79.7
InternVL3-1B+FT
84.1
92.7
93.3
91.0
Table 6: Evaluation of VQA models on our manually annotated dataset. ‘FT’ denotes fine-tuning on the proposed training data.
Figure 4: Average recall and inference speed of FIRE-CIR depending on the number n of re-ranked images, evaluated on the Fashion IQ validation set.
Figure 5: Qualitative examples of re-ranking with FIRE-CIR on the Fashion IQ dataset. We show the top-5 images (left to right) retrieved by each method (FashionBLIP-2 and FIRE-CIR). The ground-truth target image is framed in green.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: In this example, FIRE-CIR is able to accurately identify the pattern in the dresses and re-ranks the retrieved results according to their resemblance with what is described in the modification text.
Figure 7: While FashionBLIP-2 ranks highly the reference image as some of its visual aspects remain compatible with the CIR query, FIRE-CIR correctly identifies that this dress does not have a solid black color, and thus re-ranks it outside of the top-50 results. The ground-truth target image is at the 8th rank, but the 7 other top-ranked images are also compatible with the given query.
Figure 8: Similarly, visual similarity and the green design contribute to having candidate images similar to the reference one in the top-retrieved results. However, FIRE-CIR accurately promotes the ground-truth image, as its color is more coherent with what is written in the modification text.
Figure 9: Contrary to FashionBLIP-2 which focuses specifically on the “workout muscle” characteristic, FIRE-CIR gives more importance to all the features, including the orange color.