Composed Image Retrieval (CIR) aims to retrieve a target image given a reference image and a modification text. Although recent approaches leveraging Multimodal Large Language Models (MLLMs) have shown significant progress, we argue that the MLLM's compositional understanding remains underutilized. This stems from directly mapping composed queries into a retrieval embedding via contrastive supervision alone, without explicitly resolving the intended visual modifications. In this work, we introduce Factorized MLLM Embeddings (FactEmbed), which decouples compositional query comprehension from building discriminative embeddings. Rather than directly projecting the composed query into a single embedding without internal reasoning, FactEmbed employs a set of factorized latents that simulate the intended visual modifications via an intermediate target imagination objective. This process encourages the latents to ground into target semantics before forming discriminative representations for retrieval. Subsequently, to propagate these factorized representations throughout the model for retrieval, we optimize a factor-level contrastive loss while distilling the decomposed representations into a deeper attention layer of the MLLM. Finally, for efficient inference, we retain the [EOS] token to establish a global-level embedding on top of these factorized latents, while we exploit the mean-pooled factorized latents to rerank a short candidate list at negligible cost, as both are produced by the same forward pass. Extensive evaluations demonstrate that FactEmbed outperforms prior works across CIR benchmarks.
Figures & tables
Figure 1: Overview of FactEmbed. Given a composed query and a target image, FactEmbed introduces a compact set of learnable factorized latents into the input sequence. To enforce structured factorization without requiring dense annotations, we attach a decoder to an intermediate MLLM layer, which utilizes these latents as an information bottleneck to reconstruct target patch features via a target imagination objective. In addition, we apply a factor-level InfoNCE to the resulting latents {uk}k=1K to leverage the decomposed factors for retrieval, while global-level embedding z is trained with the instance-level InfoNCE objective.
Method
Refined-CIRR
Refined-FashionIQ
Recall@N
Recall subset @N
Avg.
Dress
Shirt
Top&Tee
Avg.
R@1
R@5
R@10
R@50
R@1
R@2
R@3
R@10
R@50
R@10
R@50
R@10
R@50
R@10
R@50
CoVR-2 †
56.1
–
91.1
97.9
–
–
–
–
–
–
–
–
–
–
54.2
72.9
CoLLM †
60.4
–
92.6
98.2
–
–
–
–
–
–
–
–
–
–
57.2
76.4
VLM2Vec 7B
53.39
83.50
90.96
97.89
78.16
91.65
96.72
65.78
18.59
37.41
30.57
50.21
26.98
47.61
25.38
45.08
VLM2Vec-v2 2B
59.93
86.79
92.57
98.32
81.47
93.30
97.73
70.70
35.57
60.19
53.83
71.53
49.13
70.09
46.18
67.27
Table 1: Results on Refined-CIRR and Refined-FashionIQ. † are the results from CoLLM [ 18 ] . All other baselines are reproduced using official checkpoints, except for QWEN2.5-VL and ReCALL, which were re-run from scratch due to the lack of available checkpoints. ∗ denotes the use of additional gallery image captions. “Avg.” for Refined-CIRR is computed as (R@1+Rsubset@1)/2 , while on Refined-FashionIQ, it represents the mean across all three categories. Best and second-best results are shown in bold and underlined , respectively.
Table 3
CIRR
Refined-CIRR
R@1
R@5
R@10
R@1
R@5
R@10
Full Factor
56.84
84.87
91.81
69.20
91.47
95.24
Mean Factor
54.87
83.21
90.53
67.21
90.30
94.68
Mean Factor #
55.69
84.36
91.37
68.25
91.60
95.42
[EOS]
56.12
84.58
91.61
68.84
90.96
94.93
Efficient Rerank
56.60
84.87
91.78
68.99
91.29
95.42
Table 4: Inference strategy for balancing performance and efficiency. While matching the full set of factorized latents via optimal transport yields the best performance, its memory footprint scales with the number of latents. Efficient reranking achieves comparable results to full factor matching while maintaining low overhead. “Mean Factor # ” is a variant where the instance-level contrastive loss is applied to the mean-pooled latent u~ instead of [EOS] , together with the factor-level loss. Red highlights our default configuration.
Imag
NCE
Distill
ERR
Post
CIRR test
FIQ
Refined-CIRR
Refined-FIQ
R@1
R@5
R@10
R s @1
R s @2
R@10
R@50
R@1
R@5
R@10
R s @1
R s @2
R@10
R@50
a
-
-
-
-
-
51.98
81.90
90.27
77.69
91.85
52.98
73.68
64.64
89.51
94.53
84.65
94.68
66.82
83.55
b
✓
-
-
-
-
51.45
82.89
90.36
77.64
91.76
53.98
74.26
63.72
89.56
94.27
83.73
94.76
67.98
84.47
c
✓
✓
-
-
-
52.72
83.28
90.96
78.68
92.22
55.25
75.01
65.17
89.94
94.65
84.83
94.81
68.94
84.97
d
✓
-
✓
-
-
52.53
82.84
90.82
78.15
92.02
54.69
75.36
64.92
89.74
94.53
84.14
94.78
68.40
84.88
e
✓
✓
✓
-
-
53.28
83.42
91.18
78.94
92.51
55.98
75.94
65.43
90.17
94.83
84.73
94.86
69.19
85.50
Table 5: Ablation study. Imag., NCE, and Distill represent Limag (Eq. 3 ), factor-level InfoNCE, and attention distillation (Eq. 7 ), respectively. ERR represents efficient reranking (Eq. 9 ) and Post denotes the post-training stage with augmented triplets. R s @N denotes Recall subset @N.
Figure 2: Studies on imagination layer and comparison against NCE-only baselines with different matching metrics. MS and OT denote maximum similarity and optimal transport applied to latents, respectively, without the imagination objective for factorization.
Figure 3: Qualitative analysis of target imagination. For each composed query, we visualize target DINOv2 features (DINOv2 PCA) and features imagined by our decoder from the query (Recon PCA), projected onto a joint PCA basis. Alpha Mask Argmax shows, for each spatial position p , the factorized latent with the largest decoder alpha weight, i.e . , argmaxkMˉk,p , where each color denotes a different latent. On top of the coarse layout from the target prior, fine details are recovered by the latents, jointly reconstructing the target structure.
Figure 4: Impact of layout prior in target imagination. Three rows compare factor decomposition across different layout prior configurations: (top) without a layout prior, (middle) with a 1×1 layout prior, and (bottom) with our default 2×2 layout prior. While the unconditioned model collapses into coarse global semantics, introducing the 2×2 layout prior guides the latents to isolate semantically editable parts.
Backbone
R@1
R@5
R@10
R@50
R s @1
R s @2
R s @3
Avg.
CIRR test
Qwen2.5-VL-7B
56.60
84.87
91.78
98.41
81.86
93.78
97.61
83.37
Qwen3-VL-8B
59.25
86.10
91.93
98.36
83.54
94.53
97.74
84.82
Qwen3-VL-Emb-8B
60.43
86.99
93.35
98.70
84.15
94.82
97.81
85.57
Refined CIRR
Qwen2.5-VL-7B
68.99
91.29
95.42
99.06
87.45
96.16
98.63
78.22
Qwen3-VL-8B
71.31
92.29
95.72
99.16
89.33
96.59
98.88
80.32
Qwen3-VL-Emb-8B
73.60
93.02
96.41
99.13
88.95
97.12
98.88
81.28
Table 6: FactEmbed with different MLLM backbones. “Avg.” follows Tab. 3 and Tab. 1 , i.e . , (R@5 + R s @1)/2 on CIRR test and (R@1 + R s @1)/2 on Refined-CIRR.
Figure 5: Additional Qualitative Analysis of Target Imagination.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Methods
Raw Similarity
Prototype-removed
within instance
across instances
within instance
a
Learnable Latents only (no auxiliary loss)
0.606
0.408
0.599
b
Latents with InfoNCE
0.764
0.548
0.841
c
Ours
0.586
0.426
0.548
Appendix
Table 7: Diversity analysis on factorized latents. “Within instance” denotes the mean cosine similarity across all latent pairs within the same image, whereas “Across instances” denotes the mean cosine similarity between latents of different images. Further, we report prototype-removed diversity which is computed by subtracting the dataset-wide per-latent mean vector (computed over the CIRR dataset) from each latent to isolate diversity from MLLM residual biases.
Retrieval Unit
Num Vectors
Scoring
Total
Single (bs=1)
Batched (bs=32)
Single (bs=1)
Batched (bs=32)
ms/q
ms/q
ms/q
Δ
ms/q
Δ
[EOS] only
1
0.06
0.004
353.1
–
163.8
–
+ Efficient Rerank
2
0.23
0.007
353.3
+0.17ms
163.8
+0.003ms
Full Factor
8
3.99
0.31
357.1
+3.93ms
164.1
+0.31ms
Appendix
Table 8: Per-query inference cost on CIRR validation set (2,297 gallery images). We report the number of vectors indexed per gallery image, the scoring cost (similarity matching), and the total inference latency for both single and batched queries. Total inference latency involves the scoring cost and the encoding cost of the query-side, while the cost of encoding offline gallery (target-side) is excluded. Δ indicates the latency overhead relative to the [EOS] -only baseline.
Figure 6: Impact of augmentation strategies for post-training. Starting from our model before post-training (Stage 2), we compare post-training with triplets augmented by the two-image prompting of ReCALL [ 76 ] and by our contrastive captioning. Numbers above the bars denote the absolute gains over stage 2 baseline.
Composed Image Retrieval (CIR) aims to retrieve target images based on a reference image and modified texts. However, existing methods often struggle to extract the correct semantic cues from the reference image that best reflect the user's intent under textual modification prompts, resulting in interference from irrelevant visual noise. In this paper, we propose a novel Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning (MCoT-MVS) for CIR, integrating attention-aware multi-level vision features guided by reasoning cues from a multi-modal large language model (MLLM). Specifically, we leverage an MLLM to perform chain-of-thought reasoning on the multimodal composed input, generating the retained, removed, and target-inferred texts. These textual cues subsequently guide two reference visual attention selection modules to selectively extract discriminative patch-level and instance-level semantics from the reference image. Finally, to effectively fuse these multi-granular visual cues with the modified text and the imagined target description, we design a weighted hierarchical combination module to align the composed query with target images in a unified embedding space. Extensive experiments on two CIR benchmarks, namely CIRR and FashionIQ, demonstrate that our approach consistently outperforms existing methods and achieves new state-of-the-art performance. Code and trained models are publicly released.
Xuri Ge, Chunhao Wang, Xindi Wang +3
Shandong University Jinan, China · Shandong University Qingdao, China
Composed image retrieval (CIR) aims to retrieve a desired target image from a query consisting of a reference image and a modification text. This task exhibits an unusual representational asymmetry: the modification text specifies a transition from the reference state, whereas retrieval candidates depict completed target states. This creates a representation mismatch for zero-shot methods that query pretrained vision-language spaces directly with transformation-oriented language. We study this mismatch and reformulate zero-shot composed image retrieval as target-state reconstruction followed by retrieval. We instantiate this formulation with ASAP-CIR, a training-free framework that reconstructs a static target representation using a frozen multimodal large language model (MLLM). The representation combines multiple holistic descriptions with a variable set of importance-weighted atomic semantics, thereby preserving both overall target identity and fine-grained visual constraints. Retrieval then integrates holistic state alignment, atomic constraint grounding, and calibrated target-state evidence aggregation. A controlled text-only diagnostic shows that target-side static query formulations achieve more reliable retrieval than dynamic composed query formulations, particularly when source-state semantics must be suppressed or transformed. Experiments on FashionIQ, CIRR, and CIRCO further characterize the effectiveness and limitations of this representation principle, with the clearest gains on the multi-target CIRCO benchmark. These results show that how composed intent is represented before retrieval is a consequential design choice, distinct from the choice of retrieval backbone itself.
Yihe Zhao, Songhe Feng
School of Computer Science and Technology, Beijing Jiaotong University, Beijing, 100044, China.
Zero-shot composed image retrieval (ZS-CIR) aims to retrieve a target image based on a reference image and modification text. Recent approaches commonly rely on large vision-language models to generate target image captions for retrieval. However, these captions may include redundant semantic details, introducing noise when retrieving the target image and leading to suboptimal retrieval results. To address this issue, we propose Less-is-More CIR, a two-stage training-free framework that combines Concise Retrieval and Critic Re-ranking. In the first stage, concise target-oriented captions are generated that preserve only the most discriminative modification semantics, thereby reducing semantic noise in the retrieval. In the second stage, a critic re-rank module further evaluates the retrieved candidates from the perspectives of modification consistency and reference preservation. Extensive experiments demonstrate the effectiveness and robustness of the proposed framework for the ZS-CIR. Our code is available at: https://github.com/dxlabskku/LM-CIR.
Yuqian Zheng, Hyoju Ahn, Hyeongjin Ahn +1
Department of Applied Artificial Intelligence, Sungkyunkwan University, Seoul, Republic of Korea · Department of Immersive Media Engineering, Sungkyunkwan University, Seoul, Republic of Korea · Jaume I University, Castellon, Spain