Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image, and a relative caption that specifies the desired modification. Despite the rapid development of CIR models, their performance is not well characterized by existing benchmarks, which inherently contain indeterminate queries degrading the evaluation (i.e., multiple candidate images, rather than solely the target image, meet the query criteria), and have not considered their effectiveness in the context of the multi-round system. Motivated by this, we consider improving the evaluation procedure from two aspects: 1) we introduce FISD, a Fully-Informed Semantically-Diverse benchmark, which employs generative models to precisely control the variables of reference-target image pairs, enabling a more accurate evaluation of CIR methods across six dimensions, without query ambiguity; 2) we propose an automatic multi-round agentic evaluation framework to probe the potential of the existing models in the interactive scenarios. By observing how models adapt and refine their choices over successive rounds of queries, this framework provides a more realistic appraisal of their efficacy in practical applications. Extensive experiments and comparisons prove the value of our novel evaluation on typical CIR methods.
Figures & tables
Figure 2: An overview of our FISD benchmark. The left side shows the data construction process, which includes two stages: caption generation and image generation. The right side displays examples of data for different semantic aspects, where the reference image is marked with an orange frame, the target image with a red frame, and hard negative image with a gray frame.
Cardinality
Addition
Negation
Change
Background
Complex Inst.
Average
Model
Recall@1
Recall@1
Recall@1
Recall@1
Recall@1
Recall@1
Recall@1
Pic2Word [ 39 ]
6.00
39.00
0.00
24.00
18.00
41.00
21.33
Context-I2W [ 45 ]
14.00
48.00
0.00
29.50
27.50
48.00
27.83
LinCIR [ 10 ]
5.50
52.50
1.50
43.50
36.50
55.00
32.42
TransAgg [ 30 ]
15.50
56.00
2.00
52.00
26.00
57.00
34.75
CLIP4CIR [ 4 ]
31.00
67.50
4.00
65.00
43.50
72.00
47.17
Table 1: Performance of various CIR models on FISD benchmark.
Figure 3: Overview of our automated multi-round evaluation framework. The user initially provides a reference image and the relative caption. Subsequently, the CIR model takes these inputs to generate a composed query feature, which is then stored in the history list. Next, the ranker uses the history list and all image features to determine candidate images. Finally, the selected candidate image becomes the reference image for the next round and is fed into the user simulator to generate the relative caption for the subsequent round.
Method
FashionIQ-Dress
FashionIQ-Shirt
FashionIQ-Toptee
CIRR
CIRCO
FISD
Hits@10
Hits@50
Hits@10
Hits@50
Hits@10
Hits@50
Hits@1
Hits@5
MAP@5
Hits@1
Round 1
Pic2Word [ 39 ]
20.00
40.20
26.20
43.60
27.90
47.40
23.25
51.42
7.39
21.33
Context-I2W [ 45 ]
23.10
45.30
29.70
48.60
30.60
52.90
26.96
56.59
11.96
27.83
LinCIR [ 10 ]
20.92
42.44
29.10
46.81
28.81
50.18
25.09
54.41
10.61
32.42
TransAgg [ 30 ]
30.24
51.91
34.45
53.97
38.40
59.51
38.79
69.58
11.06
34.75
Table 2: Multi-round evaluation on various state-of-the-art CIR models across a range of benchmarks.
Figure 4: Performance across various rounds on FashionIQ.
MLLM
LLM
Hits@1
Hits@5
BLIP-2
Llama3-8B
85.82
97.54
LLaVA-1.5
Llama3-8B
89.62
98.37
LLaVA-Next
Llama3-8B
89.76
98.49
LLaVA-Next
Llama2-7B
78.43
94.07
LLaVA-Next
Mistral-7B
89.45
98.64
LLaVA-Next
Llama3-8B
89.76
98.49
Table 3: Performance of various MLLMs and LLMs on CIRR.
Method
R1/(Hits@1 ↑ /Rank ↓ )
R3/(Hits@1 ↑ /Rank ↓ )
R5/(Hits@1 ↑ /Rank ↓ )
Qwen2VL
55.39 / 6.21
80.80 / 2.01
90.00 / 1.53
Llama3
55.39 / 6.21
81.37 / 1.79
89.76 / 1.45
Table 4: Comparison of various simulation methods on CIRR. R1 represents Round 1.
Method
Dataset
Initial Rank
Final Rank
Human
CIRR
189.98
12.34 ↓177.64
Human
FashionIQ
2427.12
447.96 ↓1979.16
Human
FISD
9.62
2.92 ↓6.7
Simulator
CIRR
189.98
4.12 ↓185.86
Simulator
FashionIQ
2427.12
370.78 ↓2056.34
Simulator
FISD
9.62
4.16 ↓5.46
Table 5: Performance of real users and our user simulator. Our user simulator aligns with the performance of the real users.
Method
Round 1 / Rank ↓
Round 3 / Rank ↓
Round 5 / Rank ↓
No History
6.21
2.93
2.39
Cap. Unchanged
6.21
8.56
9.64
Top-10 random
6.21
1.86
1.70
Ours
6.21
1.79
1.45
Table 6: Ablation studies on CIRR Benchmark based on SPRC model. Bold indicates the best result.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10
Figure 5: The template for cardinality relative captions.
Figure 12Figure 13Figure 14Figure 15
Method
Negation
All
R@1
R@1
Pic2Word
17.81
23.25
Context-I2W
20.55
26.96
SPRC
31.51
55.39
Appendix
Table 7: Comparison of performance on negation and overall semantics in the CIRR validation set.
Method
FashionIQ-Dress
FashionIQ-Shirt
FashionIQ-Toptee
CIRR
FISD
Recall@10
Recall@50
Recall@10
Recall@50
Recall@10
Recall@50
Recall@1
Recall@5
Recall@1
Round 1
Pic2Word [ 39 ]
20.00
40.20
26.20
43.60
27.90
47.40
23.25
51.42
21.33
Context-I2W [ 45 ]
23.10
45.30
29.70
48.60
30.60
52.90
26.96
56.59
27.83
LinCIR [ 10 ]
20.92
42.44
29.10
46.81
28.81
50.18
25.09
54.41
32.42
TransAgg [ 30 ]
30.24
51.91
34.45
53.97
38.40
59.51
38.79
69.58
34.75
Appendix
Table 8: Multi-round evaluation on various state-of-the-art CIR models across a range of benchmarks.
Figure 6: Some examples of feedback from both users and user simulators.
Figure 7: Failure cases of current CIR models on CIRR validation set. The reference image is marked with blue borders and the target image with red borders.
Figure 8: Failure cases of current CIR models on FashionIQ validation set. The reference image is marked with blue borders and the target image with red borders.
Figure 9: Some examples of our proposed benchmark, the order of the images is a reference image, target image, and hard negative image.
Figure 10: Some examples of our proposed benchmark, the order of the images is a reference image, target image, and hard negative image.