Training-Free Pseudo-Fusion for Composed Image Retrieval with Diffusion Models and Multimodal Large Language Models

Fan Xu; Luis A. Leiva

cs.CVAug 24, 2026

Training-Free Pseudo-Fusion for Composed Image Retrieval with Diffusion Models and Multimodal Large Language Models

Authors: Fan Xu, Luis A. Leiva

Organizations: Department of Computer Science University of Luxembourg

Abstract

Composed Image Retrieval (CIR) is an emerging paradigm in content-based image retrieval that enables users to formulate compositional queries by combining a reference image with an auxiliary modality, usually text-based. This approach supports fine-grained search where the target image shares structural elements with the user-provided image while incorporating the modifications specified by the auxiliary text. Conventional CIR methods rely on multimodal fusion to combine visual and textual features into a joint query embedding, which requires training modules that align composed queries with the targets. In this work, we propose PeFuse (for pseudo-fusion), a training-free framework that leverages pretrained Diffusion Models and Multimodal Large Language Models to bridge modalities via generative conversion. We introduce two novel strategies: uni-directional and bi-directional conversion, which convert CIR into four single-modality retrieval problems. These methods reformulate CIR as either intra-modal or cross-modal single-query retrieval tasks, bypassing the need for dedicated task-specific training. Extensive experiments on standard benchmarks demonstrate that converting CIR into text-to-image retrieval tasks is more effective than alternative conversion strategies, achieving competitive or superior performance compared with state-of-the-art methods, while maintaining high flexibility thanks to replaceable components of the conversion pipeline. These results highlight the effectiveness of the pseudo-fusion paradigm for zero-shot CIR. Our code is publicly available at: https://github.com/StevenXuf/PeFuse4CIR.

Figures & tables

Figure 1: Training-free pseudo-fusion framework for Composed Image Retrieval. Dashed green box indicates bi-directional conversion, while the dashed purple box is uni-directional conversion. T2I: text-to-image; T2T: text-to-text; I2T: image-to-text; I2I: image-to-image. Black arrows indicate the modality conversion flow.

Panel of Figure 2 — Algorithm 1 Training-Free Pseudo-Fusion ( PeFuse ) for Composed Image Retrieval

Method	Retrieval Model	Shirt		Dress		Toptee		Average
Method	Retrieval Model	R@10	R@50	R@10	R@50	R@10	R@50	R@10	R@50
CIReVL ∗	CLIP (ViT-B/32)	18.40	30.82	14.25	30.45	18.00	34.33	16.88	31.87
LDRE †	CLIP (ViT-B/32)	27.38	46.27	19.97	41.84	27.07	48.78	24.81	45.63
WeiMoCIR †	OpenCLIP (ViT-B/32)	29.20	47.15	26.23	46.31	34.17	55.99	29.86	49.82
ImageScope †	CLIP (ViT-B/32)	24.29	37.49	18.00	35.20	24.99	41.41	22.42	38.03
PeFuse (T $\rightarrow$ I)	CLIP	20.62	37.11	13.99	32.54	19.93	39.58	18.18	36.41

Table 1: Performance (%) on Fashion-IQ validation split using different retrieval models via PeFuse . Best and second best highlighted. ∗ : reproduced results; † : results from original papers. Training-based baselines are reported in Table 8 in Appendix B .

Method	Retrieval Model	mAP@5	mAP@10	mAP@25	mAP@50
CIReVL ∗	CLIP (ViT-B/32)	10.36	10.70	11.88	12.47
LDRE †	CLIP (ViT-B/32)	17.96	18.32	20.21	21.11
ImageScope †	CLIP (ViT-B/32)	22.36	22.19	23.03	23.83
PeFuse (T $\rightarrow$ I)	CLIP	11.12	11.47	12.86	13.61
	OpenCLIP	16.89	17.56	19.14	20.16
	SigLIP2	18.53	19.68	21.58	22.62

Table 3: Performance (%) on CIRCO test split using different retrieval models via PeFuse . Best and second best highlighted. ∗ : reproduced results; † : results from original papers. Training-based baselines are reported in Table 10 in Appendix B .

Method	Retrieval Model	Focus Attribute			Change Attribute			Focus Object			Change Object			Average
Method	Retrieval Model	R@1	R@2	R@3	R@1	R@2	R@3	R@1	R@2	R@3	R@1	R@2	R@3	R@1
CIReVL †	CLIP (ViT-B/32)	17.90	29.40	40.40	14.80	25.80	35.80	14.60	24.30	33.30	16.10	27.80	37.60	15.90
PeFuse (T $\rightarrow$ I)	CLIP	18.20	31.15	41.85	14.54	25.76	35.89	13.67	25.26	34.90	17.40	29.03	38.67	15.95
	OpenCLIP	19.55	31.95	43.70	14.73	26.28	36.51	17.96	28.27	38.06	16.73	30.97	41.33	17.22
	SigLIP2	17.00	29.35	39.65	14.63	26.70	36.17	18.27	29.34	38.27	19.59	31.58	42.24	17.37
PeFuse (I $\rightarrow$ I)	CLIP	15.90	27.15	37.50	11.03	20.36	27.60	9.18	18.88	27.35	8.16	17.60	27.65	11.07

Table 4: Performance (%) comparison on GeneCIS test split using different retrieval models via PeFuse . Best and second best highlighted. † : results from original papers. Training-based baselines are reported in Table 11 in Appendix B .

Backbone	Fashion-IQ		CIRR					CIRCO				GeneCIS
Backbone	R@10	R@50	R@1	R@2	R@5	R@10	R@50	mAP@5	mAP@10	mAP@25	mAP@50	R@1
ViT-L/14 ( laion2b_s32b_b82k )	29.49	48.37	36.17	50.07	67.13	78.72	94.17	21.69	22.99	25.07	26.14	16.85
ViT-H/14 ( laion2b_s32b_b79k )	30.40	50.09	38.55	52.02	69.49	80.29	94.29	23.78	24.78	27.10	28.24	17.50
ViT-g/14 ( laion2b_s34b_b88k )	30.15	50.12	38.41	52.15	70.15	80.27	94.58	22.62	23.93	26.39	27.42	17.39
ViT-bigG/14 ( laion2b_s39b_b160k )	30.44	49.49	40.41	54.63	71.13	81.06	94.89	25.03	26.63	29.17	30.28	17.99

Table 5: Scaling laws when using different pretrained backbones for OpenCLIP for text-to-image retrieval on each benchmark.

Figure 2: CIR performance (%) of PeFuse (T

\rightarrow

I) via the MLLM under varying values for hyperparameters on the CIRCO validation split using OpenCLIP with a ViT-B/32 backbone for text-to-image retrieval over three runs. Shaded regions indicate the standard deviation.

Figure 3: CIR performance (%) of PeFuse (I

\rightarrow

I) via Diffusion models with and without using composed descriptions from the MLLM on CIRCO validation split using OpenCLIP with a ViT-B/32 backbone for image-to-image retrieval over three runs. Shaded regions indicate the standard deviation.

Method	Generative Models	Memory (MB)	Inference Time (s)	Pipeline Time (s)	Avg mAP (%)
Modification $\rightarrow$ Target	$-$	$-$	$-$	0.48 ( $\pm 0.02$ )	7.36
Reference $\rightarrow$ Target	$-$	$-$	$-$	0.48 ( $\pm 0.02$ )	3.48
Reference+modification $\rightarrow$ Target	$-$	$-$	$-$	0.49 ( $\pm 0.02$ )	6.61
PeFuse (T $\rightarrow$ I)	Qwen2.5-VL-3B-Instruct	7162	0.69 ( $\pm$ 0.03)	1.70 ( $\pm$ 0.11)	25.15 ( $\pm$ 0.80)
	Qwen2.5-VL-7B-Instruct	15818	0.41 ( $\pm$ 0.03)	1.70 ( $\pm$ 0.14)	24.97 ( $\pm$ 0.37)
	LLaVA-1.5-7B-hf	13472	0.70 ( $\pm$ 0.04)	3.01 ( $\pm$ 0.10)	17.11 ( $\pm$ 1.09)

Table 6: Ablation and compute latency on the CIRCO validation split over three runs. For baseline performance, all retrieval tasks are deterministic, thus no standard deviation is reported; no generative models are involved, therefore no memory consumption is measured. For PeFuse conversion modes, generative inference time and the entire pipeline time are reported per query , with mean and standard deviation. “SDXL-Instr.” denotes SDXL-InstructPix2Pix .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	# of Queries	# of Candidates
Fashion-IQ (Shirt)	1940	6181
Fashion-IQ (Dress)	1859	3648
Fashion-IQ (Toptee)	1867	5261
CIRCO	800	123403
CIRR	4148	2315
GeneCIS (Focus Attribute)	2000	20000

Appendix

Table 7: Public benchmarks used in our experiments. We use the validation split for Fashion-IQ and test splits for CIRCO, CIRR, and GeneCIS.

Method	Retrieval Model	Shirt		Dress		Toptee		Average
Method	Retrieval Model	R@10	R@50	R@10	R@50	R@10	R@50	R@10	R@50
Pic2Word †	CLIP (ViT-L/14)	26.20	43.60	20.00	40.20	27.90	47.40	24.70	43.70
SEARLE-OTI ∗	CLIP (ViT-B/32)	24.43	41.39	19.85	40.72	24.85	45.47	23.05	42.53
SEARLE ∗	CLIP (ViT-B/32)	24.85	41.60	19.37	39.21	25.12	46.22	23.11	42.34
CoLLM †	CLIP (ViT-B/32)	24.90	45.10	22.90	43.80	26.40	46.80	24.80	45.20
LinCIR ∗	CLIP (ViT-L/14)	29.69	48.14	22.32	45.13	30.85	52.01	27.62	48.43

Appendix

Table 8: Performance (%) of training-based methods on Fashion-IQ validation split. Best and second best highlighted. ∗ : reproduced results; † : results from original papers.

Method	Retrieval Model	Recall					Recall ${}_{\text{subset}}$
Method	Retrieval Model	@1	@2	@5	@10	@50	@1	@2	@3
Pic2Word †	CLIP (ViT-L/14)	23.90	—	51.70	65.30	87.80	53.76	74.46	87.08
SEARLE-OTI ∗	CLIP (ViT-B/32)	23.18	34.72	52.31	66.00	89.21	52.02	74.43	86.75
SEARLE ∗	CLIP (ViT-B/32)	23.33	34.89	52.89	66.99	89.81	53.90	76.19	87.76
CoLLM †	CLIP (ViT-B/32)	28.60	—	—	71.80	92.70	—	—	—
LinCIR ∗	CLIP (ViT-L/14)	25.04	36.22	53.78	67.18	88.75	56.53	76.82	88.70

Appendix

Table 9: Performance (%) of training-based methods on CIRR test split. ∗ : reproduced results; † : results from original papers;

-

: results not available.

Method	Retrieval Model	mAP@5	mAP@10	mAP@25	mAP@50
Pic2Word ‡	CLIP (ViT-L/14)	8.72	9.51	10.64	11.29
SEARLE-OTI ∗	CLIP (ViT-B/32)	7.29	7.99	9.21	9.85
SEARLE ∗	CLIP (ViT-B/32)	9.38	9.95	11.13	11.85
CoLLM †	CLIP (ViT-B/32)	12.90	13.20	—	15.00
LinCIR ∗	CLIP (ViT-L/14)	12.33	13.13	14.56	15.46
SEARLE+CIG-XL turbo †	CLIP (ViT-B/32)	10.45	11.02	12.34	13.00

Appendix

Table 10: Performance (%) of training-based methods on CIRCO test split. ∗ : reproduced results; † : results from original papers;

-

: results not available.

Method	Retrieval Model	Focus Attribute			Change Attribute			Focus Object			Change Object			Average
Method	Retrieval Model	R@1	R@2	R@3	R@1	R@2	R@3	R@1	R@2	R@3	R@1	R@2	R@3	R@1
Pic2Word †	CLIP (ViT-L/14)	15.65	28.16	38.65	13.87	24.67	33.05	8.42	18.01	25.77	6.68	15.05	24.03	11.16
SEARLE †	CLIP (ViT-B/32)	18.90	30.60	41.20	13.00	23.80	33.70	12.20	23.00	33.30	13.60	23.80	33.30	14.40
LinCIR †	CLIP (ViT-L/14)	16.90	29.95	41.45	16.19	27.98	36.84	8.27	17.40	26.22	7.40	15.71	25.00	12.19
LinCIR+CIG-XL turbo †	CLIP (ViT-L/14)	16.80	29.70	40.90	15.91	28.88	37.45	8.37	17.35	25.10	7.86	15.46	24.29	12.24

Appendix

Table 11: Performance (%) of training-based methods on GeneCIS test split.

Backbone	Shirt		Dress		Toptee		CIRR
Backbone	R@10	R@50	R@10	R@50	R@10	R@50	R ${}_{\text{subset}}$ @1	R ${}_{\text{subset}}$ @2	R ${}_{\text{subset}}$ @3
ViT-L/14	29.54	47.01	25.23	43.73	33.69	54.37	73.49	88.82	95.08
ViT-H/14	30.41	47.22	26.90	47.71	33.90	55.33	74.41	89.23	95.33
ViT-g/14	30.52	48.87	25.12	46.05	34.82	55.44	74.15	89.57	95.45
ViT-bigG/14	31.39	47.32	25.12	45.56	34.82	55.60	75.90	89.37	95.59

Appendix

Table 12: Scaling laws on each category of Fashion-IQ and CIRR subsets using OpenCLIP for the text-to-image retrieval task.

Backbone	Focus Attribute			Change Attribute			Focus Object			Change Object
Backbone	R@1	R@2	R@3	R@1	R@2	R@3	R@1	R@2	R@3	R@1	R@2	R@3
ViT-L/14	17.85	30.30	41.90	14.25	27.27	37.12	16.63	27.50	38.21	18.67	31.02	40.61
ViT-H/14	19.10	31.35	43.50	15.58	27.18	37.97	17.50	28.16	37.09	17.81	29.59	39.54
ViT-g/14	19.25	32.05	42.10	15.77	27.94	37.83	17.35	27.50	37.24	17.19	29.69	39.39
ViT-bigG/14	19.00	31.15	42.95	16.57	28.79	39.44	17.55	27.91	37.45	18.83	30.77	40.87

Appendix

Table 13: Scaling laws on each category of GeneCIS using OpenCLIP for text-to-image task.

Panel of Figure 4 — Figure 4: Qualitative results of SDXL-InstructPix2Pix -generated images with Qwen2.5-VL-7B-generated descriptions and with raw captions from the CIRCO validation split. We use 3.0 for image guidance scale, 7.5 for guidance scale, and 30 denoising steps.

Panel of Figure 5 — Figure 5: Qualitative results of SDXL-Turbo -generated images with Qwen2.5-VL-7B-generated descriptions and with raw captions from the CIRCO validation split. We use 0.5 for strength, 0.0 for guidance scale, and 30 denoising steps.

Figure 6: Failure cases of PeFuse (T $\rightarrow$ I). None of the top-five results matches the corresponding query.

Figure 7: Failure cases of PeFuse (I $\rightarrow$ I). None of the top-five results matches the corresponding query.

Explore similar work

DiCE-CIR: Direct Composition Learning for Efficient Zero-Shot Composed Image Retrieval
Jul 6, 2026Gwang-Ho Na, Ho-Joong Kim, Seong-Whan LeeCross-Modal Alignment Compositional Representation Learning
Department of Artificial Intelligence, Korea University, Anam-dong, Seongbuk-ku, Seoul 02841, Korea
G-MIXER: Geodesic Mixup-based Implicit Semantic Expansion and Explicit Semantic Re-ranking for Zero-Shot Composed Image Retrieval
Apr 16, 2026Jiyoung Lim, Heejae Yang, Jee-Hyong LeeZero-Shot Learning Composed Image Retrieval
Sungkyunkwan University
Do Composed Image Retrieval Benchmarks Require Multimodal Composition?
May 14, 2026Matteo Attimonelli, Alessandro De Bellis, Aryo Pradipta Gema +8Benchmark Auditing Image Retrieval
Politecnico di Bari

Jul 6, 2026, cs.CVJ/K move · Enter open · S save

DiCE-CIR: Direct Composition Learning for Efficient Zero-Shot Composed Image Retrieval

Gwang-Ho Na, Ho-Joong Kim, Seong-Whan Lee

Department of Artificial Intelligence, Korea University, Anam-dong, Seongbuk-ku, Seoul 02841, Korea

Cross-Modal Alignment Compositional Representation Learning Composed Image Retrieval

Open paper AskPDF arXiv

Zero-shot composed image retrieval (ZS-CIR) aims to retrieve a target image from a multimodal query consisting of a reference image and an edit text describing the desired modification. Recent ZS-CIR studies have relied on projection-based methods that map a reference image into pseudo-word tokens in the text embedding space. However, such methods require additional projection and re-encoding steps, increasing training complexity, reducing efficiency, and introducing a discrepancy between training and inference. In this paper, we propose DiCE-CIR, a direct composition learning method that predicts composed query representations by directly composing a reference image and an edit text. To enable scalable training without manually annotated triplets, we automatically construct compositional training samples from large-scale image-caption pairs using a large language model. Based on these samples, we train a lightweight composition module with objectives that promote alignment with the target, edit-consistent semantic transformation, and retrieval discriminability. We conduct extensive experiments on ZS-CIR benchmarks and show that DiCE-CIR achieves state-of-the-art performance on CIRCO and competitive performance on CIRR while maintaining high computational efficiency.

Abstract

Explore similar work

DiCE-CIR: Direct Composition Learning for Efficient Zero-Shot Composed Image Retrieval

G-MIXER: Geodesic Mixup-based Implicit Semantic Expansion and Explicit Semantic Re-ranking for Zero-Shot Composed Image Retrieval

Do Composed Image Retrieval Benchmarks Require Multimodal Composition?