Training-Free Pseudo-Fusion for Composed Image Retrieval with Diffusion Models and Multimodal Large Language Models

Fan Xu; Luis A. Leiva

cs.CVAug 24, 2026

Training-Free Pseudo-Fusion for Composed Image Retrieval with Diffusion Models and Multimodal Large Language Models

Authors: Fan Xu, Luis A. Leiva

Organizations: Department of Computer Science University of Luxembourg

Abstract

Composed Image Retrieval (CIR) is an emerging paradigm in content-based image retrieval that enables users to formulate compositional queries by combining a reference image with an auxiliary modality, usually text-based. This approach supports fine-grained search where the target image shares structural elements with the user-provided image while incorporating the modifications specified by the auxiliary text. Conventional CIR methods rely on multimodal fusion to combine visual and textual features into a joint query embedding, which requires training modules that align composed queries with the targets. In this work, we propose PeFuse (for pseudo-fusion), a training-free framework that leverages pretrained Diffusion Models and Multimodal Large Language Models to bridge modalities via generative conversion. We introduce two novel strategies: uni-directional and bi-directional conversion, which convert CIR into four single-modality retrieval problems. These methods reformulate CIR as either intra-modal or cross-modal single-query retrieval tasks, bypassing the need for dedicated task-specific training. Extensive experiments on standard benchmarks demonstrate that converting CIR into text-to-image retrieval tasks is more effective than alternative conversion strategies, achieving competitive or superior performance compared with state-of-the-art methods, while maintaining high flexibility thanks to replaceable components of the conversion pipeline. These results highlight the effectiveness of the pseudo-fusion paradigm for zero-shot CIR. Our code is publicly available at: https://github.com/StevenXuf/PeFuse4CIR.

Figures & tables

Figure 1: Training-free pseudo-fusion framework for Composed Image Retrieval. Dashed green box indicates bi-directional conversion, while the dashed purple box is uni-directional conversion. T2I: text-to-image; T2T: text-to-text; I2T: image-to-text; I2I: image-to-image. Black arrows indicate the modality conversion flow.

Panel of Figure 2 — Algorithm 1 Training-Free Pseudo-Fusion ( PeFuse ) for Composed Image Retrieval

Method	Retrieval Model	Shirt		Dress		Toptee		Average
Method	Retrieval Model	R@10	R@50	R@10	R@50	R@10	R@50	R@10	R@50
CIReVL ∗	CLIP (ViT-B/32)	18.40	30.82	14.25	30.45	18.00	34.33	16.88	31.87
LDRE †	CLIP (ViT-B/32)	27.38	46.27	19.97	41.84	27.07	48.78	24.81	45.63
WeiMoCIR †	OpenCLIP (ViT-B/32)	29.20	47.15	26.23	46.31	34.17	55.99	29.86	49.82
ImageScope †	CLIP (ViT-B/32)	24.29	37.49	18.00	35.20	24.99	41.41	22.42	38.03
PeFuse (T $\rightarrow$ I)	CLIP	20.62	37.11	13.99	32.54	19.93	39.58	18.18	36.41

Table 1: Performance (%) on Fashion-IQ validation split using different retrieval models via PeFuse . Best and second best highlighted. ∗ : reproduced results; † : results from original papers. Training-based baselines are reported in Table 8 in Appendix B .

Method	Retrieval Model	mAP@5	mAP@10	mAP@25	mAP@50
CIReVL ∗	CLIP (ViT-B/32)	10.36	10.70	11.88	12.47
LDRE †	CLIP (ViT-B/32)	17.96	18.32	20.21	21.11
ImageScope †	CLIP (ViT-B/32)	22.36	22.19	23.03	23.83
PeFuse (T $\rightarrow$ I)	CLIP	11.12	11.47	12.86	13.61
	OpenCLIP	16.89	17.56	19.14	20.16
	SigLIP2	18.53	19.68	21.58	22.62

Table 3: Performance (%) on CIRCO test split using different retrieval models via PeFuse . Best and second best highlighted. ∗ : reproduced results; † : results from original papers. Training-based baselines are reported in Table 10 in Appendix B .

Method	Retrieval Model	Focus Attribute			Change Attribute			Focus Object			Change Object			Average
Method	Retrieval Model	R@1	R@2	R@3	R@1	R@2	R@3	R@1	R@2	R@3	R@1	R@2	R@3	R@1
CIReVL †	CLIP (ViT-B/32)	17.90	29.40	40.40	14.80	25.80	35.80	14.60	24.30	33.30	16.10	27.80	37.60	15.90
PeFuse (T $\rightarrow$ I)	CLIP	18.20	31.15	41.85	14.54	25.76	35.89	13.67	25.26	34.90	17.40	29.03	38.67	15.95
	OpenCLIP	19.55	31.95	43.70	14.73	26.28	36.51	17.96	28.27	38.06	16.73	30.97	41.33	17.22
	SigLIP2	17.00	29.35	39.65	14.63	26.70	36.17	18.27	29.34	38.27	19.59	31.58	42.24	17.37
PeFuse (I $\rightarrow$ I)	CLIP	15.90	27.15	37.50	11.03	20.36	27.60	9.18	18.88	27.35	8.16	17.60	27.65	11.07

Table 4: Performance (%) comparison on GeneCIS test split using different retrieval models via PeFuse . Best and second best highlighted. † : results from original papers. Training-based baselines are reported in Table 11 in Appendix B .

Backbone	Fashion-IQ		CIRR					CIRCO				GeneCIS
Backbone	R@10	R@50	R@1	R@2	R@5	R@10	R@50	mAP@5	mAP@10	mAP@25	mAP@50	R@1
ViT-L/14 ( laion2b_s32b_b82k )	29.49	48.37	36.17	50.07	67.13	78.72	94.17	21.69	22.99	25.07	26.14	16.85
ViT-H/14 ( laion2b_s32b_b79k )	30.40	50.09	38.55	52.02	69.49	80.29	94.29	23.78	24.78	27.10	28.24	17.50
ViT-g/14 ( laion2b_s34b_b88k )	30.15	50.12	38.41	52.15	70.15	80.27	94.58	22.62	23.93	26.39	27.42	17.39
ViT-bigG/14 ( laion2b_s39b_b160k )	30.44	49.49	40.41	54.63	71.13	81.06	94.89	25.03	26.63	29.17	30.28	17.99

Table 5: Scaling laws when using different pretrained backbones for OpenCLIP for text-to-image retrieval on each benchmark.

Figure 2: CIR performance (%) of PeFuse (T

\rightarrow

I) via the MLLM under varying values for hyperparameters on the CIRCO validation split using OpenCLIP with a ViT-B/32 backbone for text-to-image retrieval over three runs. Shaded regions indicate the standard deviation.

Figure 3: CIR performance (%) of PeFuse (I

\rightarrow

I) via Diffusion models with and without using composed descriptions from the MLLM on CIRCO validation split using OpenCLIP with a ViT-B/32 backbone for image-to-image retrieval over three runs. Shaded regions indicate the standard deviation.

Method	Generative Models	Memory (MB)	Inference Time (s)	Pipeline Time (s)	Avg mAP (%)
Modification $\rightarrow$ Target	$-$	$-$	$-$	0.48 ( $\pm 0.02$ )	7.36
Reference $\rightarrow$ Target	$-$	$-$	$-$	0.48 ( $\pm 0.02$ )	3.48
Reference+modification $\rightarrow$ Target	$-$	$-$	$-$	0.49 ( $\pm 0.02$ )	6.61
PeFuse (T $\rightarrow$ I)	Qwen2.5-VL-3B-Instruct	7162	0.69 ( $\pm$ 0.03)	1.70 ( $\pm$ 0.11)	25.15 ( $\pm$ 0.80)
	Qwen2.5-VL-7B-Instruct	15818	0.41 ( $\pm$ 0.03)	1.70 ( $\pm$ 0.14)	24.97 ( $\pm$ 0.37)
	LLaVA-1.5-7B-hf	13472	0.70 ( $\pm$ 0.04)	3.01 ( $\pm$ 0.10)	17.11 ( $\pm$ 1.09)

Table 6: Ablation and compute latency on the CIRCO validation split over three runs. For baseline performance, all retrieval tasks are deterministic, thus no standard deviation is reported; no generative models are involved, therefore no memory consumption is measured. For PeFuse conversion modes, generative inference time and the entire pipeline time are reported per query , with mean and standard deviation. “SDXL-Instr.” denotes SDXL-InstructPix2Pix .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	# of Queries	# of Candidates
Fashion-IQ (Shirt)	1940	6181
Fashion-IQ (Dress)	1859	3648
Fashion-IQ (Toptee)	1867	5261
CIRCO	800	123403
CIRR	4148	2315
GeneCIS (Focus Attribute)	2000	20000

Appendix

Table 7: Public benchmarks used in our experiments. We use the validation split for Fashion-IQ and test splits for CIRCO, CIRR, and GeneCIS.

Method	Retrieval Model	Shirt		Dress		Toptee		Average
Method	Retrieval Model	R@10	R@50	R@10	R@50	R@10	R@50	R@10	R@50
Pic2Word †	CLIP (ViT-L/14)	26.20	43.60	20.00	40.20	27.90	47.40	24.70	43.70
SEARLE-OTI ∗	CLIP (ViT-B/32)	24.43	41.39	19.85	40.72	24.85	45.47	23.05	42.53
SEARLE ∗	CLIP (ViT-B/32)	24.85	41.60	19.37	39.21	25.12	46.22	23.11	42.34
CoLLM †	CLIP (ViT-B/32)	24.90	45.10	22.90	43.80	26.40	46.80	24.80	45.20
LinCIR ∗	CLIP (ViT-L/14)	29.69	48.14	22.32	45.13	30.85	52.01	27.62	48.43

Appendix

Table 8: Performance (%) of training-based methods on Fashion-IQ validation split. Best and second best highlighted. ∗ : reproduced results; † : results from original papers.

Method	Retrieval Model	Recall					Recall ${}_{\text{subset}}$
Method	Retrieval Model	@1	@2	@5	@10	@50	@1	@2	@3
Pic2Word †	CLIP (ViT-L/14)	23.90	—	51.70	65.30	87.80	53.76	74.46	87.08
SEARLE-OTI ∗	CLIP (ViT-B/32)	23.18	34.72	52.31	66.00	89.21	52.02	74.43	86.75
SEARLE ∗	CLIP (ViT-B/32)	23.33	34.89	52.89	66.99	89.81	53.90	76.19	87.76
CoLLM †	CLIP (ViT-B/32)	28.60	—	—	71.80	92.70	—	—	—
LinCIR ∗	CLIP (ViT-L/14)	25.04	36.22	53.78	67.18	88.75	56.53	76.82	88.70

Appendix

Table 9: Performance (%) of training-based methods on CIRR test split. ∗ : reproduced results; † : results from original papers;

-

: results not available.

Method	Retrieval Model	mAP@5	mAP@10	mAP@25	mAP@50
Pic2Word ‡	CLIP (ViT-L/14)	8.72	9.51	10.64	11.29
SEARLE-OTI ∗	CLIP (ViT-B/32)	7.29	7.99	9.21	9.85
SEARLE ∗	CLIP (ViT-B/32)	9.38	9.95	11.13	11.85
CoLLM †	CLIP (ViT-B/32)	12.90	13.20	—	15.00
LinCIR ∗	CLIP (ViT-L/14)	12.33	13.13	14.56	15.46
SEARLE+CIG-XL turbo †	CLIP (ViT-B/32)	10.45	11.02	12.34	13.00

Appendix

Table 10: Performance (%) of training-based methods on CIRCO test split. ∗ : reproduced results; † : results from original papers;

-

: results not available.

Method	Retrieval Model	Focus Attribute			Change Attribute			Focus Object			Change Object			Average
Method	Retrieval Model	R@1	R@2	R@3	R@1	R@2	R@3	R@1	R@2	R@3	R@1	R@2	R@3	R@1
Pic2Word †	CLIP (ViT-L/14)	15.65	28.16	38.65	13.87	24.67	33.05	8.42	18.01	25.77	6.68	15.05	24.03	11.16
SEARLE †	CLIP (ViT-B/32)	18.90	30.60	41.20	13.00	23.80	33.70	12.20	23.00	33.30	13.60	23.80	33.30	14.40
LinCIR †	CLIP (ViT-L/14)	16.90	29.95	41.45	16.19	27.98	36.84	8.27	17.40	26.22	7.40	15.71	25.00	12.19
LinCIR+CIG-XL turbo †	CLIP (ViT-L/14)	16.80	29.70	40.90	15.91	28.88	37.45	8.37	17.35	25.10	7.86	15.46	24.29	12.24

Appendix

Table 11: Performance (%) of training-based methods on GeneCIS test split.

Backbone	Shirt		Dress		Toptee		CIRR
Backbone	R@10	R@50	R@10	R@50	R@10	R@50	R ${}_{\text{subset}}$ @1	R ${}_{\text{subset}}$ @2	R ${}_{\text{subset}}$ @3
ViT-L/14	29.54	47.01	25.23	43.73	33.69	54.37	73.49	88.82	95.08
ViT-H/14	30.41	47.22	26.90	47.71	33.90	55.33	74.41	89.23	95.33
ViT-g/14	30.52	48.87	25.12	46.05	34.82	55.44	74.15	89.57	95.45
ViT-bigG/14	31.39	47.32	25.12	45.56	34.82	55.60	75.90	89.37	95.59

Appendix

Table 12: Scaling laws on each category of Fashion-IQ and CIRR subsets using OpenCLIP for the text-to-image retrieval task.

Backbone	Focus Attribute			Change Attribute			Focus Object			Change Object
Backbone	R@1	R@2	R@3	R@1	R@2	R@3	R@1	R@2	R@3	R@1	R@2	R@3
ViT-L/14	17.85	30.30	41.90	14.25	27.27	37.12	16.63	27.50	38.21	18.67	31.02	40.61
ViT-H/14	19.10	31.35	43.50	15.58	27.18	37.97	17.50	28.16	37.09	17.81	29.59	39.54
ViT-g/14	19.25	32.05	42.10	15.77	27.94	37.83	17.35	27.50	37.24	17.19	29.69	39.39
ViT-bigG/14	19.00	31.15	42.95	16.57	28.79	39.44	17.55	27.91	37.45	18.83	30.77	40.87

Appendix

Table 13: Scaling laws on each category of GeneCIS using OpenCLIP for text-to-image task.

Panel of Figure 4 — Figure 4: Qualitative results of SDXL-InstructPix2Pix -generated images with Qwen2.5-VL-7B-generated descriptions and with raw captions from the CIRCO validation split. We use 3.0 for image guidance scale, 7.5 for guidance scale, and 30 denoising steps.

Panel of Figure 5 — Figure 5: Qualitative results of SDXL-Turbo -generated images with Qwen2.5-VL-7B-generated descriptions and with raw captions from the CIRCO validation split. We use 0.5 for strength, 0.0 for guidance scale, and 30 denoising steps.

Figure 6: Failure cases of PeFuse (T $\rightarrow$ I). None of the top-five results matches the corresponding query.

Figure 7: Failure cases of PeFuse (I $\rightarrow$ I). None of the top-five results matches the corresponding query.

Explore similar work

Jul 6, 2026cs.CV

DiCE-CIR: Direct Composition Learning for Efficient Zero-Shot Composed Image Retrieval

Zero-shot composed image retrieval (ZS-CIR) aims to retrieve a target image from a multimodal query consisting of a reference image and an edit text describing the desired modification. Recent ZS-CIR studies have relied on projection-based methods that map a reference image into pseudo-word tokens in the text embedding space. However, such methods require additional projection and re-encoding steps, increasing training complexity, reducing efficiency, and introducing a discrepancy between training and inference. In this paper, we propose DiCE-CIR, a direct composition learning method that predicts composed query representations by directly composing a reference image and an edit text. To enable scalable training without manually annotated triplets, we automatically construct compositional training samples from large-scale image-caption pairs using a large language model. Based on these samples, we train a lightweight composition module with objectives that promote alignment with the target, edit-consistent semantic transformation, and retrieval discriminability. We conduct extensive experiments on ZS-CIR benchmarks and show that DiCE-CIR achieves state-of-the-art performance on CIRCO and competitive performance on CIRR while maintaining high computational efficiency.

Gwang-Ho Na, Ho-Joong Kim, Seong-Whan Lee

Department of Artificial Intelligence, Korea University, Anam-dong, Seongbuk-ku, Seoul 02841, Korea

Cross-Modal Alignment Compositional Representation Learning Composed Image Retrieval

Ask PDF

Apr 16, 2026cs.CV

G-MIXER: Geodesic Mixup-based Implicit Semantic Expansion and Explicit Semantic Re-ranking for Zero-Shot Composed Image Retrieval

Composed Image Retrieval (CIR) aims to retrieve target images by integrating a reference image with a corresponding modification text. CIR requires jointly considering the explicit semantics specified in the query and the implicit semantics embedded within its bi-modal composition. Recent training-free Zero-Shot CIR (ZS-CIR) methods leverage Multimodal Large Language Models (MLLMs) to generate detailed target descriptions, converting the implicit information into explicit textual expressions. However, these methods rely heavily on the textual modality and fail to capture the fuzzy retrieval nature that requires considering diverse combinations of candidates. This leads to reduced diversity and accuracy in retrieval results. To address this limitation, we propose a novel training-free method, Geodesic Mixup-based Implicit semantic eXpansion and Explicit semantic Re-ranking for ZS-CIR (G-MIXER). G-MIXER constructs composed query features that reflect the implicit semantics of reference image-text pairs through geodesic mixup over a range of mixup ratios, and builds a diverse candidate set. The generated candidates are then re-ranked using explicit semantics derived from MLLMs, improving both retrieval diversity and accuracy. Our proposed G-MIXER achieves state-of-the-art performance across multiple ZS-CIR benchmarks, effectively handling both implicit and explicit semantics without additional training. Our code will be available at https://github.com/maya0395/gmixer.

Jiyoung Lim, Heejae Yang, Jee-Hyong Lee

Sungkyunkwan University

Zero-Shot Learning Composed Image Retrieval

Ask PDF

May 14, 2026cs.CV

Do Composed Image Retrieval Benchmarks Require Multimodal Composition?

Composed Image Retrieval (CIR) is a multimodal retrieval task where a query consists of a reference image and a textual modification, and the goal is to retrieve a target image satisfying both. In principle, strong performance on CIR benchmarks is assumed to require multimodal composition, i.e., combining complementary information from reference image and textual modification. In this work, we show that this assumption does not always hold. Across four widely used CIR benchmarks and eleven Generalist Multimodal Embedding models, a large fraction of queries can be solved using a single modality (from 32.2% to 83.6%), revealing pervasive unimodal shortcuts. Thus, high CIR performance can arise from unimodal signals rather than true multimodal composition. To better understand this issue, we perform a two-stage audit. First, we identify shortcut-solvable queries through cross-model analysis. Second, we conduct human validation on 4,741 shortcut-free queries, of which only 1,689 are well-formed, with common issues including ambiguous edits and mismatched targets. Re-evaluating models on this validated subset reveals qualitatively different behaviour: queries can no longer be solved with a single modality, and successful retrieval requires combining both inputs. While accuracy decreases, reliance on multimodal information increases. Overall, current CIR benchmarks conflate shortcut-solvable, noisy, and genuinely compositional queries, leading to an overestimation of model capability in multimodal composition.

Matteo Attimonelli, Alessandro De Bellis, Aryo Pradipta Gema +8

Politecnico di Bari

Benchmark Auditing Image Retrieval Shortcut Learning Multimodal Model Evaluation Composed Image Retrieval

Ask PDF