Factorized MLLM Embeddings for Composed Image Retrieval

WonJun Moon; Seongchan Kim; Dahun Kim; Seungryong Kim

cs.CVNov 3, 2026

Factorized MLLM Embeddings for Composed Image Retrieval

Authors: WonJun Moon, Seongchan Kim, Dahun Kim, Seungryong Kim

Organizations: KAIST AI · Google DeepMind

Abstract

Composed Image Retrieval (CIR) aims to retrieve a target image given a reference image and a modification text. Although recent approaches leveraging Multimodal Large Language Models (MLLMs) have shown significant progress, we argue that the MLLM's compositional understanding remains underutilized. This stems from directly mapping composed queries into a retrieval embedding via contrastive supervision alone, without explicitly resolving the intended visual modifications. In this work, we introduce Factorized MLLM Embeddings (FactEmbed), which decouples compositional query comprehension from building discriminative embeddings. Rather than directly projecting the composed query into a single embedding without internal reasoning, FactEmbed employs a set of factorized latents that simulate the intended visual modifications via an intermediate target imagination objective. This process encourages the latents to ground into target semantics before forming discriminative representations for retrieval. Subsequently, to propagate these factorized representations throughout the model for retrieval, we optimize a factor-level contrastive loss while distilling the decomposed representations into a deeper attention layer of the MLLM. Finally, for efficient inference, we retain the [EOS] token to establish a global-level embedding on top of these factorized latents, while we exploit the mean-pooled factorized latents to rerank a short candidate list at negligible cost, as both are produced by the same forward pass. Extensive evaluations demonstrate that FactEmbed outperforms prior works across CIR benchmarks.

Figures & tables

Figure 1: Overview of FactEmbed. Given a composed query and a target image, FactEmbed introduces a compact set of learnable factorized latents into the input sequence. To enforce structured factorization without requiring dense annotations, we attach a decoder to an intermediate MLLM layer, which utilizes these latents as an information bottleneck to reconstruct target patch features via a target imagination objective. In addition, we apply a factor-level InfoNCE to the resulting latents $\{u_{k}\}_{k=1}^{K}$ to leverage the decomposed factors for retrieval, while global-level embedding $z$ is trained with the instance-level InfoNCE objective.

Method	Refined-CIRR								Refined-FashionIQ
	Recall@N				Recall subset @N			Avg.	Dress		Shirt		Top&Tee		Avg.
	R@1	R@5	R@10	R@50	R@1	R@2	R@3	Avg.	R@10	R@50	R@10	R@50	R@10	R@50	R@10	R@50
CoVR-2 †	56.1	–	91.1	97.9	–	–	–	–	–	–	–	–	–	–	54.2	72.9
CoLLM †	60.4	–	92.6	98.2	–	–	–	–	–	–	–	–	–	–	57.2	76.4
VLM2Vec 7B	53.39	83.50	90.96	97.89	78.16	91.65	96.72	65.78	18.59	37.41	30.57	50.21	26.98	47.61	25.38	45.08
VLM2Vec-v2 2B	59.93	86.79	92.57	98.32	81.47	93.30	97.73	70.70	35.57	60.19	53.83	71.53	49.13	70.09	46.18	67.27

Table 1: Results on Refined-CIRR and Refined-FashionIQ.

\dagger

are the results from CoLLM [ 18 ] . All other baselines are reproduced using official checkpoints, except for QWEN2.5-VL and ReCALL, which were re-run from scratch due to the lack of available checkpoints.

\ast

denotes the use of additional gallery image captions. “Avg.” for Refined-CIRR is computed as

(R@1+R_{\mathrm{subset}}@1)/2

, while on Refined-FashionIQ, it represents the mean across all three categories. Best and second-best results are shown in bold and underlined , respectively.

Table 3

	CIRR			Refined-CIRR
	R@1	R@5	R@10	R@1	R@5	R@10
Full Factor	56.84	84.87	91.81	69.20	91.47	95.24
Mean Factor	54.87	83.21	90.53	67.21	90.30	94.68
Mean Factor #	55.69	84.36	91.37	68.25	91.60	95.42
[EOS]	56.12	84.58	91.61	68.84	90.96	94.93
Efficient Rerank	56.60	84.87	91.78	68.99	91.29	95.42

Table 4: Inference strategy for balancing performance and efficiency. While matching the full set of factorized latents via optimal transport yields the best performance, its memory footprint scales with the number of latents. Efficient reranking achieves comparable results to full factor matching while maintaining low overhead. “Mean Factor # ” is a variant where the instance-level contrastive loss is applied to the mean-pooled latent

\tilde{u}

instead of [EOS] , together with the factor-level loss. Red highlights our default configuration.

	Imag	NCE	Distill	ERR	Post	CIRR test					FIQ		Refined-CIRR					Refined-FIQ
	Imag	NCE	Distill	ERR	Post	R@1	R@5	R@10	R s @1	R s @2	R@10	R@50	R@1	R@5	R@10	R s @1	R s @2	R@10	R@50
a	-	-	-	-	-	51.98	81.90	90.27	77.69	91.85	52.98	73.68	64.64	89.51	94.53	84.65	94.68	66.82	83.55
b	✓	-	-	-	-	51.45	82.89	90.36	77.64	91.76	53.98	74.26	63.72	89.56	94.27	83.73	94.76	67.98	84.47
c	✓	✓	-	-	-	52.72	83.28	90.96	78.68	92.22	55.25	75.01	65.17	89.94	94.65	84.83	94.81	68.94	84.97
d	✓	-	✓	-	-	52.53	82.84	90.82	78.15	92.02	54.69	75.36	64.92	89.74	94.53	84.14	94.78	68.40	84.88
e	✓	✓	✓	-	-	53.28	83.42	91.18	78.94	92.51	55.98	75.94	65.43	90.17	94.83	84.73	94.86	69.19	85.50

Table 5: Ablation study. Imag., NCE, and Distill represent

\mathcal{L}_{\mathrm{imag}}

(Eq. 3 ), factor-level InfoNCE, and attention distillation (Eq. 7 ), respectively. ERR represents efficient reranking (Eq. 9 ) and Post denotes the post-training stage with augmented triplets. R s @N denotes Recall subset @N.

Panel of Figure 2 — Figure 2: Studies on imagination layer and comparison against NCE-only baselines with different matching metrics. MS and OT denote maximum similarity and optimal transport applied to latents, respectively, without the imagination objective for factorization.

Figure 3: Qualitative analysis of target imagination. For each composed query, we visualize target DINOv2 features (DINOv2 PCA) and features imagined by our decoder from the query (Recon PCA), projected onto a joint PCA basis. Alpha Mask Argmax shows, for each spatial position $p$ , the factorized latent with the largest decoder alpha weight, i.e . , $\arg\max_{k}\bar{\mathcal{M}}_{k,p}$ , where each color denotes a different latent. On top of the coarse layout from the target prior, fine details are recovered by the latents, jointly reconstructing the target structure.

Figure 4: Impact of layout prior in target imagination. Three rows compare factor decomposition across different layout prior configurations: (top) without a layout prior, (middle) with a $1\times 1$ layout prior, and (bottom) with our default $2\times 2$ layout prior. While the unconditioned model collapses into coarse global semantics, introducing the $2\times 2$ layout prior guides the latents to isolate semantically editable parts.

	Backbone	R@1	R@5	R@10	R@50	R s @1	R s @2	R s @3	Avg.
CIRR test	Qwen2.5-VL-7B	56.60	84.87	91.78	98.41	81.86	93.78	97.61	83.37
	Qwen3-VL-8B	59.25	86.10	91.93	98.36	83.54	94.53	97.74	84.82
	Qwen3-VL-Emb-8B	60.43	86.99	93.35	98.70	84.15	94.82	97.81	85.57
Refined CIRR	Qwen2.5-VL-7B	68.99	91.29	95.42	99.06	87.45	96.16	98.63	78.22
	Qwen3-VL-8B	71.31	92.29	95.72	99.16	89.33	96.59	98.88	80.32
	Qwen3-VL-Emb-8B	73.60	93.02	96.41	99.13	88.95	97.12	98.88	81.28

Table 6: FactEmbed with different MLLM backbones. “Avg.” follows Tab. 3 and Tab. 1 , i.e . , (R@5 + R s @1)/2 on CIRR test and (R@1 + R s @1)/2 on Refined-CIRR.

Figure 5: Additional Qualitative Analysis of Target Imagination.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Methods	Raw Similarity		Prototype-removed
	Methods	within instance	across instances	within instance
a	Learnable Latents only (no auxiliary loss)	0.606	0.408	0.599
b	Latents with InfoNCE	0.764	0.548	0.841
c	Ours	0.586	0.426	0.548

Appendix

Table 7: Diversity analysis on factorized latents. “Within instance” denotes the mean cosine similarity across all latent pairs within the same image, whereas “Across instances” denotes the mean cosine similarity between latents of different images. Further, we report prototype-removed diversity which is computed by subtracting the dataset-wide per-latent mean vector (computed over the CIRR dataset) from each latent to isolate diversity from MLLM residual biases.

Retrieval Unit	Num Vectors	Scoring		Total
		Single (bs=1)	Batched (bs=32)	Single (bs=1)		Batched (bs=32)
		ms/q	ms/q	ms/q	$\Delta$	ms/q	$\Delta$
[EOS] only	1	0.06	0.004	353.1	–	163.8	–
+ Efficient Rerank	2	0.23	0.007	353.3	+0.17ms	163.8	+0.003ms
Full Factor	8	3.99	0.31	357.1	+3.93ms	164.1	+0.31ms

Appendix

Table 8: Per-query inference cost on CIRR validation set (2,297 gallery images). We report the number of vectors indexed per gallery image, the scoring cost (similarity matching), and the total inference latency for both single and batched queries. Total inference latency involves the scoring cost and the encoding cost of the query-side, while the cost of encoding offline gallery (target-side) is excluded.

\Delta

indicates the latency overhead relative to the [EOS] -only baseline.

Panel of Figure 6 — Figure 6: Impact of augmentation strategies for post-training. Starting from our model before post-training (Stage 2), we compare post-training with triplets augmented by the two-image prompting of ReCALL [ 76 ] and by our contrastive captioning. Numbers above the bars denote the absolute gains over stage 2 baseline.

Composed Image Retrieval

Composed image retrieval finds images that match a reference image together with a text instruction describing the desired change.

See all 66 papers

MCoT-MVS: Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning for Composed Image Retrieval
Mar 18, 2026Xuri Ge, Chunhao Wang, Xindi Wang +3Composed Image Retrieval VLM Reasoning
Shandong University Jinan, China · Shandong University Qingdao, China
From Transformation to Target State: Rethinking Query Representation for Zero-Shot Composed Image Retrieval
Oct 5, 2026Yihe Zhao, Songhe FengCross-Modal Alignment Composed Image Retrieval
School of Computer Science and Technology, Beijing Jiaotong University, Beijing, 100044, China.
Less is More: Concise Retrieval and Critic-based Re-ranking for Composed Image Retrieval
Oct 20, 2026Yuqian Zheng, Hyoju Ahn, Hyeongjin Ahn +1Composed Image Retrieval Image Retrieval
Department of Applied Artificial Intelligence, Sungkyunkwan University, Seoul, Republic of Korea · Department of Immersive Media Engineering, Sungkyunkwan University, Seoul, Republic of Korea · Jaume I University, Castellon, Spain

Mar 18, 2026J/K move · Enter open · S save

MCoT-MVS: Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning for Composed Image Retrieval

Xuri Ge, Chunhao Wang, Xindi Wang, Zheyun Qin+2

Shandong University Jinan, China · Shandong University Qingdao, China

Composed Image Retrieval VLM Reasoning

Open paper AskPDF arXiv

Composed Image Retrieval (CIR) aims to retrieve target images based on a reference image and modified texts. However, existing methods often struggle to extract the correct semantic cues from the reference image that best reflect the user's intent under textual modification prompts, resulting in interference from irrelevant visual noise. In this paper, we propose a novel Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning (MCoT-MVS) for CIR, integrating attention-aware multi-level vision features guided by reasoning cues from a multi-modal large language model (MLLM). Specifically, we leverage an MLLM to perform chain-of-thought reasoning on the multimodal composed input, generating the retained, removed, and target-inferred texts. These textual cues subsequently guide two reference visual attention selection modules to selectively extract discriminative patch-level and instance-level semantics from the reference image. Finally, to effectively fuse these multi-granular visual cues with the modified text and the imagined target description, we design a weighted hierarchical combination module to align the composed query with target images in a unified embedding space. Extensive experiments on two CIR benchmarks, namely CIRR and FashionIQ, demonstrate that our approach consistently outperforms existing methods and achieves new state-of-the-art performance. Code and trained models are publicly released.

Abstract

Composed Image Retrieval

Explore similar work

MCoT-MVS: Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning for Composed Image Retrieval

From Transformation to Target State: Rethinking Query Representation for Zero-Shot Composed Image Retrieval

Less is More: Concise Retrieval and Critic-based Re-ranking for Composed Image Retrieval