Beyond Semantic Search: Towards Referential Anchoring in Composed Image Retrieval
Authors: Yuxin Yang, Yinan Zhou, Yuxin Chen, Ziqi Zhang, Zongyang Ma, Chunfeng Yuan, Bing Li, Jun Gao, Weiming Hu
Organizations: Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xi’an Jiaotong University · Tencent Inc. · PeopleAI Inc. · HelloGroup Inc. · ShanghaiTech University
Composed Image Retrieval (CIR) has demonstrated significant potential by enabling flexible multimodal queries that combine a reference image and modification text. However, CIR inherently prioritizes semantic matching, struggling to reliably retrieve a user-specified instance across contexts. In practice, emphasizing concrete instance fidelity over broad semantics is often more consequential. In this work, we propose Object-Anchored Composed Image Retrieval (OACIR), a novel fine-grained retrieval task that mandates strict instance-level consistency. To advance research on this task, we construct OACIRR (OACIR on Real-world images), the first large-scale, multi-domain benchmark comprising over 160K quadruples and four challenging candidate galleries enriched with hard-negative instance distractors. Each quadruple augments the compositional query with a bounding box that visually anchors the object in the reference image, providing a precise and flexible way to ensure instance preservation. To address the OACIR task, we propose AdaFocal, a framework featuring a Context-Aware Attention Modulator that adaptively intensifies attention within the specified instance region, dynamically balancing focus between the anchored instance and the broader compositional context. Extensive experiments demonstrate that AdaFocal substantially outperforms existing compositional retrieval models, particularly in maintaining instance-level fidelity, thereby establishing a robust baseline for this challenging task while opening new directions for more flexible, instance-aware retrieval systems.
Figures & tables
Figure 2: The multi-stage construction pipeline for the OACIRR dataset.
Dataset
Publication
# Samples
Splits
Data Type
Avg Length of
Instance
Instance
Visual
Contextual
Multi-
Modification Text
Consistency
Distractors
Grounding
Modification Text
Domain
CIRR [ 31 ]
ICCV 2021
36.6K
train, eval
real-world
11.3
✗
✗
✗
✗
✓
FashionIQ [ 45 ]
CVPR 2021
30.1K
train, eval
real-world
5.3
✗
✓
✗
✗
✗
CIRCO [ 6 ]
ICCV 2023
1.0K
eval
real-world
8.2
✗
✗
✗
✗
✓
InstructPix2Pix [ 8 ]
CVPR 2023
454K
train, eval
synthetic
9.4
✗
✗
✗
✗
✓
LaSCo [ 21 ]
AAAI 2024
389K
train
synthetic
5.9
✗
–
✗
✗
✓
Table 1: Comparative analysis of existing Multimodal Image Retrieval datasets.
Figure 3: Instance distribution of the OACIRR benchmark.
Figure 4: Overall architecture of our proposed AdaFocal framework.
Domain
Method
Pretraining Data
Fashion
Car
Product
Landmark
Avg.
RID@1
R@1
R@5
RID@1
R@1
R@5
RID@1
R@1
R@5
RID@1
R@1
R@5
UMR
UniIR-CLIPSF [ 43 ]
M-BEIR [ 43 ]
17.33
12.26
24.76
32.67
16.95
41.89
33.71
18.22
40.10
29.47
15.51
43.24
27.18
UniIR-BLIPFF [ 43 ]
28.53
22.41
39.63
37.21
19.97
46.51
37.76
20.98
43.19
31.71
17.14
52.12
33.10
LamRA-Ret [ 30 ]
M-BEIR + NLI [ 36 ]
27.45
21.63
37.10
61.03
35.44
74.51
69.45
39.53
70.25
58.64
32.58
68.74
49.70
MM-Embed [ 28 ]
M-BEIR + MTEB [ 35 ]
41.38
34.55
52.50
53.21
30.06
62.80
71.03
41.47
71.15
78.85
38.88
79.32
54.60
GME ( 2B ) [ 52 ]
UMRB [ 52 ]
38.13
32.14
51.50
58.84
31.60
66.03
76.89
44.11
74.20
73.86
38.99
75.61
55.16
Table 2: Quantitative comparison on the OACIRR benchmark. “Avg.” represents the average results across all evaluation metrics.
CAAM
OACIRR Benchmark
CRM
Probe Tokens
RID@1
R@1
R@5
Avg.
Baseline (w/o CAAM)
77.74
58.39
88.61
74.91
Average Pooling
Frozen
79.70
59.84
89.62
76.39
Learnable
79.83
59.57
89.54
76.31
MLP
Frozen
80.51
60.55
90.15
77.07
Learnable
81.10
61.10
90.40
77.53
Table 3: Ablation study on the architecture of the CAAM .
Figure 5: Ablation study on the Modulation Scalar β .
Figure 6: Qualitative comparison of our AdaFocal and the Baseline on the OACIRR benchmark. Green boxes indicate the ground-truth target, yellow boxes indicate instance-correct but semantically incorrect results, and all other retrieved images are marked with red boxes .
Subset
Filtering Threshold
τvalid
τhigh
τcentric
τcount
Fashion
8
0.92
0.88
3
Car
10
0.88
0.85
2
Product
20
0.88
0.85
2
Landmark
15
0.90
0.88
3
Table 4: Filtering Thresholds for each OACIRR subset.
Statistic
Number
Percentage
Total Annotated Quadruples
127,166
- Fashion
12,874
10.1%
- Car
12,728
10.0%
- Product
75,616
59.5%
- Landmark
25,948
20.4%
Total Unique Images
39,495
Table 5: Statistics of OACIRR Training Dataset.
Statistic
Number
Percentage
Total Annotated Quadruples
33,449
- Fashion
3,606
10.8%
- Car
3,586
10.7%
- Product
21,046
62.9%
- Landmark
5,211
15.6%
Total Unique Images
26,595
Table 6: Statistics of OACIRR Evaluation Benchmark.
Domain
Method
Pretraining Data
Fashion
Car
Product
Landmark
Avg.
RID@1
R@1
R@5
RID@1
R@1
R@5
RID@1
R@1
R@5
RID@1
R@1
R@5
Setting 1: Instance-as-Textual Adaptation
UMR
LamRA-Ret [ 30 ]
M-BEIR + NLI
25.93
20.54
36.26
58.13
33.87
72.10
67.27
36.64
67.51
57.05
32.06
67.99
47.95
MM-Embed [ 28 ]
M-BEIR + MTEB
38.05
32.70
50.69
51.37
29.62
61.74
66.68
36.73
65.49
75.95
37.75
78.53
52.11
GME ( 2B ) [ 52 ]
UMRB
37.10
31.45
51.33
55.91
30.37
63.94
75.91
40.90
72.39
72.65
38.76
74.46
53.76
GME ( 7B ) [ 52 ]
44.54
38.33
59.51
58.73
35.05
70.91
81.87
53.42
82.97
76.20
46.82
82.27
60.89
Table 7: Quantitative comparison on the OACIRR benchmark under different evaluation settings and OACIR-specific baselines. “Avg.” represents the average results across all evaluation metrics. The best result is highlighted in bold , and the second best is underlined . Baseline denotes the standard CIR baseline, Baseline† denotes the ROI-cropped baseline, and SPRC† denotes plug-and-play CIR baseline.
Method
Pretraining Data
Pretraining Scale
FashionIQ
CIRR
CIRCO
Avg@10
Avg@50
R@1
R@5
Rs@1
Avg.
mAP@5
mAP@10
mAP@25
CASE [ 21 ]
LaSCo + CoCo
389 K
–
–
35.40
65.78
64.29
65.04
–
–
–
CoVR-BLIP [ 39 ]
WebVid-CoVR
1,644 K
27.70
44.63
38.48
66.70
69.28
67.99
21.43
22.33
24.47
CompoDiff [ 15 ]
ST18M + LAION-2B
18,000 K
39.02
51.71
26.71
55.14
64.54
59.84
15.33
17.71
19.45
CoAlign (ViT-G) [ 22 ]
CIRHS
535 K
39.22
60.08
41.08
71.11
70.80
70.96
21.60
23.38
25.98
Baseline (ViT-L)
37.82
59.57
41.59
72.36
72.06
72.21
21.51
23.14
25.47
Table 8: Zero-shot cross-task generalization on standard CIR benchmarks.
Setting
Method
Fashion
Car
Product
Landmark
Avg.
RID@1
R@1
R@5
RID@1
R@1
R@5
RID@1
R@1
R@5
RID@1
R@1
R@5
SPRC [ 4 ]
48.73
40.51
62.84
66.86
41.49
78.98
62.79
42.68
71.72
59.49
38.55
71.23
57.16
Cross-Domain
AdaFocal
61.15
50.25
71.54
74.26
45.65
85.81
67.84
45.53
74.68
61.58
40.66
71.94
62.57
SPRC [ 4 ]
65.25
58.51
80.89
72.87
49.82
89.57
86.05
70.61
93.68
76.32
56.04
89.00
74.05
Full Finetuning
AdaFocal
77.15
65.31
86.88
78.42
53.63
92.22
91.86
74.11
95.39
82.92
58.47
91.63
79.00
Table 9: Cross-domain generalization on the OACIRR benchmark.
Bounding Box
Fashion
Car
Product
Landmark
Avg.
IoU
Perturbation
RID@1
R@1
R@5
RID@1
R@1
R@5
RID@1
R@1
R@5
RID@1
R@1
R@5
1.00
Original
77.15
65.31
86.88
78.42
53.63
92.22
91.86
74.11
95.39
82.92
58.47
91.63
79.00
0.80
Scale
77.05
65.24
86.82
78.26
53.54
92.16
91.86
74.11
95.35
82.83
58.41
91.60
78.93
0.50
Scale + Shift
75.16
63.24
85.55
77.07
52.61
91.54
91.20
73.44
94.83
81.66
57.66
90.96
77.91
NaN
w/o Bounding Box
69.07
58.76
81.44
74.59
49.78
89.46
87.48
69.53
93.66
79.80
55.49
89.87
74.91
Table 10: Robustness of AdaFocal to Scale and Shift perturbations of bounding boxes on the OACIRR benchmark.
Modulation Output
Fashion
Car
Product
Landmark
Avg.
RID@1
R@1
R@5
RID@1
R@1
R@5
RID@1
R@1
R@5
RID@1
R@1
R@5
Scalar ( β )
77.15
65.31
86.88
78.42
53.63
92.22
91.86
74.11
95.39
82.92
58.47
91.63
79.00
Vector ( β )
74.60
65.25
85.94
77.32
53.33
92.19
91.56
73.13
94.92
82.80
58.96
91.77
78.48
Table 11: Ablation study on the modulation output design of the CAAM .
CAAM
OACIRR Benchmark
# Self-Attention Layers
RID@1
R@1
R@5
Avg.
1
81.38
62.39
90.54
78.10
2
82.59
62.88
91.53
79.00
3
82.31
62.75
91.42
78.83
4
82.02
62.51
91.24
78.59
Table 12: Ablation study on the number of self-attention layers .
CAAM
OACIRR Benchmark
# Probe Tokens
RID@1
R@1
R@5
Avg.
2
81.92
63.15
91.49
78.85
4
82.38
62.41
91.15
78.65
8
82.59
62.88
91.53
79.00
16
82.46
62.94
91.45
78.95
32
82.21
62.90
91.37
78.83
Table 13: Ablation study on the number of probe tokens .
Figure 7: A curated collage of representative instances from the OACIRR benchmark.
Figure 8: Qualitative comparison of our AdaFocal and the Baseline on the OACIRR benchmark. Green boxes indicate the ground-truth target, yellow boxes indicate instance-correct but semantically incorrect results, and all other retrieved images are marked with red boxes .
Composed image retrieval (CIR) aims to retrieve images that satisfy a user-specified modification while preserving relevant visual content from a reference image. Collecting target images for this purpose is costly, motivating zero-shot CIR methods that instead use target captions as supervision. However, target captions may omit source details that should be preserved. We therefore propose, Preserve-and-Compose Training, which complements target-caption supervision with visual evidence from the source image. PACT learns from image--text--text (ITT) triplets without target images or gallery updates, aligning composed queries with target captions while preserving source evidence through visual supervision. We further introduce Chord scoring, which combines target similarity with source-relative directional agreement in the frozen image space. Results across four ZS-CIR benchmarks show that combining target-caption supervision with source-image evidence leads to strong retrieval performance across datasets, backbone scales, and external galleries. The code is available on https://github.com/sehyunkwon/PACT.
Composed Image Retrieval (CIR) aims to retrieve target images based on a reference image and modified texts. However, existing methods often struggle to extract the correct semantic cues from the reference image that best reflect the user's intent under textual modification prompts, resulting in interference from irrelevant visual noise. In this paper, we propose a novel Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning (MCoT-MVS) for CIR, integrating attention-aware multi-level vision features guided by reasoning cues from a multi-modal large language model (MLLM). Specifically, we leverage an MLLM to perform chain-of-thought reasoning on the multimodal composed input, generating the retained, removed, and target-inferred texts. These textual cues subsequently guide two reference visual attention selection modules to selectively extract discriminative patch-level and instance-level semantics from the reference image. Finally, to effectively fuse these multi-granular visual cues with the modified text and the imagined target description, we design a weighted hierarchical combination module to align the composed query with target images in a unified embedding space. Extensive experiments on two CIR benchmarks, namely CIRR and FashionIQ, demonstrate that our approach consistently outperforms existing methods and achieves new state-of-the-art performance. Code and trained models are publicly released.
Xuri Ge, Chunhao Wang, Xindi Wang +3
Shandong University Jinan, China · Shandong University Qingdao, China
Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image, and a relative caption that specifies the desired modification. Despite the rapid development of CIR models, their performance is not well characterized by existing benchmarks, which inherently contain indeterminate queries degrading the evaluation (i.e., multiple candidate images, rather than solely the target image, meet the query criteria), and have not considered their effectiveness in the context of the multi-round system. Motivated by this, we consider improving the evaluation procedure from two aspects: 1) we introduce FISD, a Fully-Informed Semantically-Diverse benchmark, which employs generative models to precisely control the variables of reference-target image pairs, enabling a more accurate evaluation of CIR methods across six dimensions, without query ambiguity; 2) we propose an automatic multi-round agentic evaluation framework to probe the potential of the existing models in the interactive scenarios. By observing how models adapt and refine their choices over successive rounds of queries, this framework provides a more realistic appraisal of their efficacy in practical applications. Extensive experiments and comparisons prove the value of our novel evaluation on typical CIR methods.
Yikun Liu, Jiangchao Yao, Weidi Xie +1
School of Artificial Intelligence, Shanghai Jiao Tong University, China · CMIC, Shanghai Jiao Tong University, China