Beyond Semantic Search: Towards Referential Anchoring in Composed Image Retrieval

Yuxin Yang; Yinan Zhou; Yuxin Chen; Ziqi Zhang; Zongyang Ma; Chunfeng Yuan; Bing Li; Jun Gao

cs.CVApr 7, 2026

Beyond Semantic Search: Towards Referential Anchoring in Composed Image Retrieval

Authors: Yuxin Yang, Yinan Zhou, Yuxin Chen, Ziqi Zhang, Zongyang Ma, Chunfeng Yuan, Bing Li, Jun Gao,

Organizations: Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xi’an Jiaotong University · Tencent Inc. ·

Abstract

Composed Image Retrieval (CIR) has demonstrated significant potential by enabling flexible multimodal queries that combine a reference image and modification text. However, CIR inherently prioritizes semantic matching, struggling to reliably retrieve a user-specified instance across contexts. In practice, emphasizing concrete instance fidelity over broad semantics is often more consequential. In this work, we propose Object-Anchored Composed Image Retrieval (OACIR), a novel fine-grained retrieval task that mandates strict instance-level consistency. To advance research on this task, we construct OACIRR (OACIR on Real-world images), the first large-scale, multi-domain benchmark comprising over 160K quadruples and four challenging candidate galleries enriched with hard-negative instance distractors. Each quadruple augments the compositional query with a bounding box that visually anchors the object in the reference image, providing a precise and flexible way to ensure instance preservation. To address the OACIR task, we propose AdaFocal, a framework featuring a Context-Aware Attention Modulator that adaptively intensifies attention within the specified instance region, dynamically balancing focus between the anchored instance and the broader compositional context. Extensive experiments demonstrate that AdaFocal substantially outperforms existing compositional retrieval models, particularly in maintaining instance-level fidelity, thereby establishing a robust baseline for this challenging task while opening new directions for more flexible, instance-aware retrieval systems.

Figures & tables

Figure 2: The multi-stage construction pipeline for the OACIRR dataset.

Dataset	Publication	# Samples	Splits	Data Type	Avg Length of	Instance	Instance	Visual	Contextual	Multi-
Dataset	Publication	# Samples	Splits	Data Type	Modification Text	Consistency	Distractors	Grounding	Modification Text	Domain
CIRR [ 31 ]	ICCV 2021	36.6K	train, eval	real-world	11.3	✗	✗	✗	✗	✓
FashionIQ [ 45 ]	CVPR 2021	30.1K	train, eval	real-world	5.3	✗	✓	✗	✗	✗
CIRCO [ 6 ]	ICCV 2023	1.0K	eval	real-world	8.2	✗	✗	✗	✗	✓
InstructPix2Pix [ 8 ]	CVPR 2023	454K	train, eval	synthetic	9.4	✗	✗	✗	✗	✓
LaSCo [ 21 ]	AAAI 2024	389K	train	synthetic	5.9	✗	–	✗	✗	✓

Table 1: Comparative analysis of existing Multimodal Image Retrieval datasets.

Figure 3: Instance distribution of the OACIRR benchmark.

Figure 4: Overall architecture of our proposed AdaFocal framework.

Domain	Method	Pretraining Data	Fashion			Car			Product			Landmark			Avg.
Domain	Method	Pretraining Data	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	Avg.
UMR	$\text{UniIR-CLIP}_{\textit{SF}}$ [ 43 ]	M-BEIR [ 43 ]	17.33	12.26	24.76	32.67	16.95	41.89	33.71	18.22	40.10	29.47	15.51	43.24	27.18
	$\text{UniIR-BLIP}_{\textit{FF}}$ [ 43 ]	M-BEIR [ 43 ]	28.53	22.41	39.63	37.21	19.97	46.51	37.76	20.98	43.19	31.71	17.14	52.12	33.10
	LamRA-Ret [ 30 ]	M-BEIR + NLI [ 36 ]	27.45	21.63	37.10	61.03	35.44	74.51	69.45	39.53	70.25	58.64	32.58	68.74	49.70
	MM-Embed [ 28 ]	M-BEIR + MTEB [ 35 ]	41.38	34.55	52.50	53.21	30.06	62.80	71.03	41.47	71.15	78.85	38.88	79.32	54.60
	GME ( 2B ) [ 52 ]	UMRB [ 52 ]	38.13	32.14	51.50	58.84	31.60	66.03	76.89	44.11	74.20	73.86	38.99	75.61	55.16

Table 2: Quantitative comparison on the OACIRR benchmark. “Avg.” represents the average results across all evaluation metrics.

CAAM		OACIRR Benchmark
CRM	Probe Tokens	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	Avg.
Baseline (w/o CAAM)		77.74	58.39	88.61	74.91
Average Pooling	Frozen	79.70	59.84	89.62	76.39
Average Pooling	Learnable	79.83	59.57	89.54	76.31
MLP	Frozen	80.51	60.55	90.15	77.07
MLP	Learnable	81.10	61.10	90.40	77.53

Table 3: Ablation study on the architecture of the CAAM .

Figure 5: Ablation study on the Modulation Scalar $\beta$ .

Figure 6: Qualitative comparison of our AdaFocal and the Baseline on the OACIRR benchmark. Green boxes indicate the ground-truth target, yellow boxes indicate instance-correct but semantically incorrect results, and all other retrieved images are marked with red boxes .

Subset	Filtering Threshold
Subset	$\tau_{valid}$	$\tau_{high}$	$\tau_{centric}$	$\tau_{count}$
Fashion	8	0.92	0.88	3
Car	10	0.88	0.85	2
Product	20	0.88	0.85	2
Landmark	15	0.90	0.88	3

Table 4: Filtering Thresholds for each OACIRR subset.

Statistic	Number	Percentage
Total Annotated Quadruples	127,166
- Fashion	12,874	10.1%
- Car	12,728	10.0%
- Product	75,616	59.5%
- Landmark	25,948	20.4%
Total Unique Images	39,495

Table 5: Statistics of OACIRR Training Dataset.

Statistic	Number	Percentage
Total Annotated Quadruples	33,449
- Fashion	3,606	10.8%
- Car	3,586	10.7%
- Product	21,046	62.9%
- Landmark	5,211	15.6%
Total Unique Images	26,595

Table 6: Statistics of OACIRR Evaluation Benchmark.

Domain	Method	Pretraining Data	Fashion			Car			Product			Landmark			Avg.
Domain	Method	Pretraining Data	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	Avg.
Setting 1: Instance-as-Textual Adaptation
UMR	LamRA-Ret [ 30 ]	M-BEIR + NLI	25.93	20.54	36.26	58.13	33.87	72.10	67.27	36.64	67.51	57.05	32.06	67.99	47.95
	MM-Embed [ 28 ]	M-BEIR + MTEB	38.05	32.70	50.69	51.37	29.62	61.74	66.68	36.73	65.49	75.95	37.75	78.53	52.11
	GME ( 2B ) [ 52 ]	UMRB	37.10	31.45	51.33	55.91	30.37	63.94	75.91	40.90	72.39	72.65	38.76	74.46	53.76
	GME ( 7B ) [ 52 ]	UMRB	44.54	38.33	59.51	58.73	35.05	70.91	81.87	53.42	82.97	76.20	46.82	82.27	60.89

Table 7: Quantitative comparison on the OACIRR benchmark under different evaluation settings and OACIR-specific baselines. “Avg.” represents the average results across all evaluation metrics. The best result is highlighted in bold , and the second best is underlined . Baseline denotes the standard CIR baseline,

\text{Baseline}^{\dagger}

denotes the ROI-cropped baseline, and

\text{SPRC}^{\dagger}

denotes plug-and-play CIR baseline.

Method	Pretraining Data	Pretraining Scale	FashionIQ		CIRR				CIRCO
Method	Pretraining Data	Pretraining Scale	Avg@10	Avg@50	R@1	R@5	$\text{R}_{\text{s}}\!\text{@1}$	Avg.	mAP@5	mAP@10	mAP@25
CASE [ 21 ]	LaSCo + CoCo	389 K	–	–	35.40	65.78	64.29	65.04	–	–	–
CoVR-BLIP [ 39 ]	WebVid-CoVR	1,644 K	27.70	44.63	38.48	66.70	69.28	67.99	21.43	22.33	24.47
CompoDiff [ 15 ]	ST18M + LAION-2B	18,000 K	39.02	51.71	26.71	55.14	64.54	59.84	15.33	17.71	19.45
CoAlign (ViT-G) [ 22 ]	CIRHS	535 K	39.22	60.08	41.08	71.11	70.80	70.96	21.60	23.38	25.98
Baseline (ViT-L)			37.82	59.57	41.59	72.36	72.06	72.21	21.51	23.14	25.47

Table 8: Zero-shot cross-task generalization on standard CIR benchmarks.

Setting	Method	Fashion			Car			Product			Landmark			Avg.
Setting	Method	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	Avg.
	SPRC [ 4 ]	48.73	40.51	62.84	66.86	41.49	78.98	62.79	42.68	71.72	59.49	38.55	71.23	57.16
Cross-Domain	AdaFocal	61.15	50.25	71.54	74.26	45.65	85.81	67.84	45.53	74.68	61.58	40.66	71.94	62.57
	SPRC [ 4 ]	65.25	58.51	80.89	72.87	49.82	89.57	86.05	70.61	93.68	76.32	56.04	89.00	74.05
Full Finetuning	AdaFocal	77.15	65.31	86.88	78.42	53.63	92.22	91.86	74.11	95.39	82.92	58.47	91.63	79.00

Table 9: Cross-domain generalization on the OACIRR benchmark.

Bounding Box		Fashion			Car			Product			Landmark			Avg.
IoU	Perturbation	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	Avg.
1.00	Original	77.15	65.31	86.88	78.42	53.63	92.22	91.86	74.11	95.39	82.92	58.47	91.63	79.00
0.80	Scale	77.05	65.24	86.82	78.26	53.54	92.16	91.86	74.11	95.35	82.83	58.41	91.60	78.93
0.50	Scale + Shift	75.16	63.24	85.55	77.07	52.61	91.54	91.20	73.44	94.83	81.66	57.66	90.96	77.91
NaN	w/o Bounding Box	69.07	58.76	81.44	74.59	49.78	89.46	87.48	69.53	93.66	79.80	55.49	89.87	74.91

Table 10: Robustness of AdaFocal to Scale and Shift perturbations of bounding boxes on the OACIRR benchmark.

Modulation Output	Fashion			Car			Product			Landmark			Avg.
Modulation Output	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	Avg.
Scalar ( $\beta$ )	77.15	65.31	86.88	78.42	53.63	92.22	91.86	74.11	95.39	82.92	58.47	91.63	79.00
Vector ( $\vec{\beta}$ )	74.60	65.25	85.94	77.32	53.33	92.19	91.56	73.13	94.92	82.80	58.96	91.77	78.48

Table 11: Ablation study on the modulation output design of the CAAM .

CAAM	OACIRR Benchmark
# Self-Attention Layers	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	Avg.
1	81.38	62.39	90.54	78.10
2	82.59	62.88	91.53	79.00
3	82.31	62.75	91.42	78.83
4	82.02	62.51	91.24	78.59

Table 12: Ablation study on the number of self-attention layers .

CAAM	OACIRR Benchmark
# Probe Tokens	$\text{R}_{\text{ID}}\!\text{@1}$	R@1	R@5	Avg.
2	81.92	63.15	91.49	78.85
4	82.38	62.41	91.15	78.65
8	82.59	62.88	91.53	79.00
16	82.46	62.94	91.45	78.95
32	82.21	62.90	91.37	78.83

Table 13: Ablation study on the number of probe tokens .

Figure 7: A curated collage of representative instances from the OACIRR benchmark.

Figure 8: Qualitative comparison of our AdaFocal and the Baseline on the OACIRR benchmark. Green boxes indicate the ground-truth target, yellow boxes indicate instance-correct but semantically incorrect results, and all other retrieved images are marked with red boxes .

Composed Image Retrieval

Composed image retrieval finds images that match a reference image together with a text instruction describing the desired change.

See all 65 papers

Preserve-and-Compose Training for Composed Image Retrieval
Sep 25, 2026Sehyun KwonComposed Image Retrieval Compositional Representation Learning
Hanyang University ERICA
MCoT-MVS: Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning for Composed Image Retrieval
Mar 18, 2026Xuri Ge, Chunhao Wang, Xindi Wang +3Composed Image Retrieval VLM Reasoning
Shandong University Jinan, China · Shandong University Qingdao, China
A Sanity Check on Composed Image Retrieval
Apr 14, 2026Yikun Liu, Jiangchao Yao, Weidi Xie +1Image Retrieval Composed Image Retrieval
School of Artificial Intelligence, Shanghai Jiao Tong University, China · CMIC, Shanghai Jiao Tong University, China

Sep 25, 2026J/K move · Enter open · S save

Preserve-and-Compose Training for Composed Image Retrieval

Sehyun Kwon

Hanyang University ERICA

Composed Image Retrieval Compositional Representation Learning Cross-Modal Alignment

Open paper AskPDF arXiv

Composed image retrieval (CIR) aims to retrieve images that satisfy a user-specified modification while preserving relevant visual content from a reference image. Collecting target images for this purpose is costly, motivating zero-shot CIR methods that instead use target captions as supervision. However, target captions may omit source details that should be preserved. We therefore propose, Preserve-and-Compose Training, which complements target-caption supervision with visual evidence from the source image. PACT learns from image--text--text (ITT) triplets without target images or gallery updates, aligning composed queries with target captions while preserving source evidence through visual supervision. We further introduce Chord scoring, which combines target similarity with source-relative directional agreement in the frozen image space. Results across four ZS-CIR benchmarks show that combining target-caption supervision with source-image evidence leads to strong retrieval performance across datasets, backbone scales, and external galleries. The code is available on https://github.com/sehyunkwon/PACT.

Abstract

Composed Image Retrieval

Explore similar work

Preserve-and-Compose Training for Composed Image Retrieval

MCoT-MVS: Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning for Composed Image Retrieval

A Sanity Check on Composed Image Retrieval