Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image, and a relative caption that specifies the desired modification. Despite the rapid development of CIR models, their performance is not well characterized by existing benchmarks, which inherently contain indeterminate queries degrading the evaluation (i.e., multiple candidate images, rather than solely the target image, meet the query criteria), and have not considered their effectiveness in the context of the multi-round system. Motivated by this, we consider improving the evaluation procedure from two aspects: 1) we introduce FISD, a Fully-Informed Semantically-Diverse benchmark, which employs generative models to precisely control the variables of reference-target image pairs, enabling a more accurate evaluation of CIR methods across six dimensions, without query ambiguity; 2) we propose an automatic multi-round agentic evaluation framework to probe the potential of the existing models in the interactive scenarios. By observing how models adapt and refine their choices over successive rounds of queries, this framework provides a more realistic appraisal of their efficacy in practical applications. Extensive experiments and comparisons prove the value of our novel evaluation on typical CIR methods.
Figures & tables
Figure 2: An overview of our FISD benchmark. The left side shows the data construction process, which includes two stages: caption generation and image generation. The right side displays examples of data for different semantic aspects, where the reference image is marked with an orange frame, the target image with a red frame, and hard negative image with a gray frame.
Cardinality
Addition
Negation
Change
Background
Complex Inst.
Average
Model
Recall@1
Recall@1
Recall@1
Recall@1
Recall@1
Recall@1
Recall@1
Pic2Word [ 39 ]
6.00
39.00
0.00
24.00
18.00
41.00
21.33
Context-I2W [ 45 ]
14.00
48.00
0.00
29.50
27.50
48.00
27.83
LinCIR [ 10 ]
5.50
52.50
1.50
43.50
36.50
55.00
32.42
TransAgg [ 30 ]
15.50
56.00
2.00
52.00
26.00
57.00
34.75
CLIP4CIR [ 4 ]
31.00
67.50
4.00
65.00
43.50
72.00
47.17
Table 1: Performance of various CIR models on FISD benchmark.
Figure 3: Overview of our automated multi-round evaluation framework. The user initially provides a reference image and the relative caption. Subsequently, the CIR model takes these inputs to generate a composed query feature, which is then stored in the history list. Next, the ranker uses the history list and all image features to determine candidate images. Finally, the selected candidate image becomes the reference image for the next round and is fed into the user simulator to generate the relative caption for the subsequent round.
Method
FashionIQ-Dress
FashionIQ-Shirt
FashionIQ-Toptee
CIRR
CIRCO
FISD
Hits@10
Hits@50
Hits@10
Hits@50
Hits@10
Hits@50
Hits@1
Hits@5
MAP@5
Hits@1
Round 1
Pic2Word [ 39 ]
20.00
40.20
26.20
43.60
27.90
47.40
23.25
51.42
7.39
21.33
Context-I2W [ 45 ]
23.10
45.30
29.70
48.60
30.60
52.90
26.96
56.59
11.96
27.83
LinCIR [ 10 ]
20.92
42.44
29.10
46.81
28.81
50.18
25.09
54.41
10.61
32.42
TransAgg [ 30 ]
30.24
51.91
34.45
53.97
38.40
59.51
38.79
69.58
11.06
34.75
Table 2: Multi-round evaluation on various state-of-the-art CIR models across a range of benchmarks.
Figure 4: Performance across various rounds on FashionIQ.
MLLM
LLM
Hits@1
Hits@5
BLIP-2
Llama3-8B
85.82
97.54
LLaVA-1.5
Llama3-8B
89.62
98.37
LLaVA-Next
Llama3-8B
89.76
98.49
LLaVA-Next
Llama2-7B
78.43
94.07
LLaVA-Next
Mistral-7B
89.45
98.64
LLaVA-Next
Llama3-8B
89.76
98.49
Table 3: Performance of various MLLMs and LLMs on CIRR.
Method
R1/(Hits@1 ↑ /Rank ↓ )
R3/(Hits@1 ↑ /Rank ↓ )
R5/(Hits@1 ↑ /Rank ↓ )
Qwen2VL
55.39 / 6.21
80.80 / 2.01
90.00 / 1.53
Llama3
55.39 / 6.21
81.37 / 1.79
89.76 / 1.45
Table 4: Comparison of various simulation methods on CIRR. R1 represents Round 1.
Method
Dataset
Initial Rank
Final Rank
Human
CIRR
189.98
12.34 ↓177.64
Human
FashionIQ
2427.12
447.96 ↓1979.16
Human
FISD
9.62
2.92 ↓6.7
Simulator
CIRR
189.98
4.12 ↓185.86
Simulator
FashionIQ
2427.12
370.78 ↓2056.34
Simulator
FISD
9.62
4.16 ↓5.46
Table 5: Performance of real users and our user simulator. Our user simulator aligns with the performance of the real users.
Method
Round 1 / Rank ↓
Round 3 / Rank ↓
Round 5 / Rank ↓
No History
6.21
2.93
2.39
Cap. Unchanged
6.21
8.56
9.64
Top-10 random
6.21
1.86
1.70
Ours
6.21
1.79
1.45
Table 6: Ablation studies on CIRR Benchmark based on SPRC model. Bold indicates the best result.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10
Figure 5: The template for cardinality relative captions.
Figure 12Figure 13Figure 14Figure 15
Method
Negation
All
R@1
R@1
Pic2Word
17.81
23.25
Context-I2W
20.55
26.96
SPRC
31.51
55.39
Appendix
Table 7: Comparison of performance on negation and overall semantics in the CIRR validation set.
Method
FashionIQ-Dress
FashionIQ-Shirt
FashionIQ-Toptee
CIRR
FISD
Recall@10
Recall@50
Recall@10
Recall@50
Recall@10
Recall@50
Recall@1
Recall@5
Recall@1
Round 1
Pic2Word [ 39 ]
20.00
40.20
26.20
43.60
27.90
47.40
23.25
51.42
21.33
Context-I2W [ 45 ]
23.10
45.30
29.70
48.60
30.60
52.90
26.96
56.59
27.83
LinCIR [ 10 ]
20.92
42.44
29.10
46.81
28.81
50.18
25.09
54.41
32.42
TransAgg [ 30 ]
30.24
51.91
34.45
53.97
38.40
59.51
38.79
69.58
34.75
Appendix
Table 8: Multi-round evaluation on various state-of-the-art CIR models across a range of benchmarks.
Figure 6: Some examples of feedback from both users and user simulators.
Figure 7: Failure cases of current CIR models on CIRR validation set. The reference image is marked with blue borders and the target image with red borders.
Figure 8: Failure cases of current CIR models on FashionIQ validation set. The reference image is marked with blue borders and the target image with red borders.
Figure 9: Some examples of our proposed benchmark, the order of the images is a reference image, target image, and hard negative image.
Figure 10: Some examples of our proposed benchmark, the order of the images is a reference image, target image, and hard negative image.
Composed Image Retrieval (CIR) has made significant progress, yet current benchmarks are limited to single ground-truth answers and lack the annotations needed to evaluate false positive avoidance, robustness and multi-image reasoning. We present PinPoint, a comprehensive real world benchmark with 7,635 queries and 329K relevance judgments across 23 query categories. PinPoint advances the field by providing: (1) multiple correct answers (averaging 9.1 per query) (2) explicit hard negatives, (3) six instruction paraphrases per query for robustness testing, (4) multi-image composition support (13.4% of queries), and (5) demographic metadata for fairness evaluation. Based on our analysis of 20+ methods across 4 different major paradigms, we uncover three significant drawbacks: The best methods while achieving mAP@10 of 28.5%, still retrieves irrelevant results (hard negatives) 9% of the time. The best models also exhibit 25.1% performance variation across paraphrases, indicating significant potential for enhancing current CIR techniques. Multi-image queries performs 40 to 70% worse across different methods. To overcome these new issues uncovered by our evaluation framework, we propose a training-free reranking method based on an off-the-shelf MLLM that can be applied to any existing system to bridge the gap. We release the complete dataset, including all images, queries, annotations, retrieval index, and benchmarking code.
Composed Image Retrieval (CIR) is a multimodal retrieval task where a query consists of a reference image and a textual modification, and the goal is to retrieve a target image satisfying both. In principle, strong performance on CIR benchmarks is assumed to require multimodal composition, i.e., combining complementary information from reference image and textual modification. In this work, we show that this assumption does not always hold. Across four widely used CIR benchmarks and eleven Generalist Multimodal Embedding models, a large fraction of queries can be solved using a single modality (from 32.2% to 83.6%), revealing pervasive unimodal shortcuts. Thus, high CIR performance can arise from unimodal signals rather than true multimodal composition. To better understand this issue, we perform a two-stage audit. First, we identify shortcut-solvable queries through cross-model analysis. Second, we conduct human validation on 4,741 shortcut-free queries, of which only 1,689 are well-formed, with common issues including ambiguous edits and mismatched targets. Re-evaluating models on this validated subset reveals qualitatively different behaviour: queries can no longer be solved with a single modality, and successful retrieval requires combining both inputs. While accuracy decreases, reliance on multimodal information increases. Overall, current CIR benchmarks conflate shortcut-solvable, noisy, and genuinely compositional queries, leading to an overestimation of model capability in multimodal composition.
Matteo Attimonelli, Alessandro De Bellis, Aryo Pradipta Gema +8
Composed image retrieval (CIR) searches a corpus with a reference image and a text describing how to modify it. Despite rapid progress from triplet-trained compositors to zero-shot and generative methods, essentially all systems share one assumption: that a query maps to a single target, scored by Recall@K against one annotation. We argue this is fundamentally at odds with the task. A query such as make it more formal does not name an image but a region of the corpus, and which member the user intends is genuinely underdetermined. This underspecification is the root of the well-known false-negative problem and leaves current models unable to tell a precise query from an ambiguous one. We reframe CIR as calibrated intent resolution under uncertainty: a retriever is wrapped in a conformal prediction layer that returns a candidate set with a coverage guarantee and whose size is a principled measure of ambiguity; when the set is large, an expected-information-gain policy asks the single most useful clarifying question, drawn from interpretable ambiguity axes, and the set contracts. We introduce AmbiCIR, a benchmark and human-validated user simulator that revive the dormant auxiliary and dialogue annotations of CIRR and extend the multiple-positive setting of CIRCO. Across open-domain and fashion benchmarks our method matches single-turn state of the art, confirming calibrated resolution is cost-free on precise queries, while reaching the intended target in a fraction of the interaction budget required by naive conversational baselines, and it is the first to report valid coverage and calibration for the task.