CaptionFuse: Training-Free Composed Product Retrieval Via VLM Intent Captioning and Weighted Embedding Fusion

Zhao Gao; Mina Metias; Mohamed Noordeen Alaudeen; Bassel Saleh

cs.CVOct 19, 2026

CaptionFuse: Training-Free Composed Product Retrieval Via VLM Intent Captioning and Weighted Embedding Fusion

Authors: Zhao Gao, Mina Metias, Mohamed Noordeen Alaudeen, Bassel Saleh

Organizations: AWS GenAI Innovation Center

Abstract

Composed Image Retrieval (CIR) retrieves products matching a reference image modified by natural language---a core capability for eCommerce search. Existing training-free methods fuse CLIP embeddings via arithmetic, but we identify a fundamental limitation: user text modifications are attribute-sparse---a user who types make it red with shorter sleeves'' specifies what to change but not what to keep (the floral pattern, the V-neck, the fit-and-flare silhouette). We introduce language-space fusion: a Vision-Language Model (VLM) jointly interprets the reference image and modification to generate a visually grounded product-title caption---e.g., Red floral fit-and-flare dress with short sleeves and V-neck''---that fills in the missing attributes from the image. Our method, CaptionFuse, requires no task-specific training and is model-agnostic. A controlled experiment confirms that attribute content--- not linguistic style---drives the gains: imperative and descriptive captions with identical attributes achieve equal R@10, while both outperform raw modifications by +6.5 points.

Figures & tables

Figure 1: Composed Image Retrieval examples across three benchmarks. Given a reference image and a text modification, the goal is to retrieve the target. FashionIQ tests fashion attribute changes; CIRR tests open-domain object modifications; CIRCO tests complex multi-aspect scene reasoning over a 123K image gallery.

Figure 2: CaptionFuse vs. WeiMoCIR on FashionIQ (dress). Each row: reference image, text modification with VLM caption (blue box), ground-truth target, and top-1 retrieval from each method. Green/orange borders = target in top 5; red = miss. The VLM fills in attributes the user left unsaid (e.g., sleeveless, opaque), enabling correct retrieval.

Panel of Figure 3 — Figure 3: CaptionFuse pipeline. The VLM generates a product-title caption fusing the reference image with the text modification. The caption and image are independently encoded, then combined with weighted fusion for retrieval. The encoder is swappable (CLIP, Nova 2, or Titan).

	FashionIQ							CIRR			CIRCO
Method	Dress		Shirt		Toptee		Avg	R@1	R@5	R@10	R@5	R@10	R@25	R@50	mAP
	R@10	R@50	R@10	R@50	R@10	R@50	R@10
Training-free baselines (CLIP ViT-L/14)
Pic2Word	20.0	40.2	26.2	43.6	27.9	47.4	24.7	23.9	51.7	65.3	6.8	13.0	24.0	34.8	11.6
LinCIR	20.9	42.0	29.1	46.8	28.8	49.2	26.3	25.0	53.3	66.7	9.5	17.1	28.4	40.2	13.2
WeiMoCIR	22.5	44.3	30.8	49.5	31.2	52.1	28.2	26.8	55.1	68.2	11.4	17.9	31.0	41.3	13.9

Table 1: Main results. R@K = Recall@K (%). Top: all methods use CLIP ViT-L/14 (apples-to-apples). Bottom: CaptionFuse with different embedding backbones (model-agnosticism). † Different embedding space (1024-dim).

Configuration	Dress	Shirt	Toptee	Avg
Fusion components (full val ⋆ )
CaptionFuse (image + caption)	22.4	27.2	24.8	24.8
Caption-only ( $\alpha{=}0$ )	20.8	25.4	23.6	23.3
No Caption (image + raw text)	17.6	24.1	24.8	22.2
Image-only ( $\alpha{=}1$ )	8.9	13.5	12.8	11.8
Fusion weight $\alpha$ (image weight, full val ⋆ )

Table 2: Key ablations on FashionIQ val. All use Nova Pro VLM and v2 prompts unless noted. R@10 (%). ⋆ Full val (6,016 queries); other rows use 200 queries/category for efficiency.

v1 output (verbose):

“A shiny silver fit and flare dress with short sleeves instead of the blue and white patterned three-quarter sleeve bodycon dress shown in the image .”

v2 output (product-title):

“Shiny Silver Fit and Flare Dress with Short Sleeves and V-Neck”

Table 3: Prompt comparison. Reference: blue patterned dress. Modification: “is shiny and silver with shorter sleeves and fit and flare.”

		FashionIQ	CIRR	CIRCO
Backbone	Dim	Avg R@10	R@1 / R@10	mAP / R@10
CLIP ViT-L/14	768	27.2	14.0 / 63.0	22.4 / 29.4
Nova 2	1024	24.6	18.7 / 66.3	24.8 / 30.6
Titan	1024	34.8	23.9 / 75.5	31.7 / 40.5

Table 4: Backbone comparison: same method, different encoders. R@10 (%) unless noted. All use CaptionFuse with identical captions and

\alpha{=}0.15

Figure 4: Recall@10 vs. image weight

\alpha

. Performance peaks at

\alpha\leq 0.2

Figure 5: Embedding geometry. Left: caption queries have higher target similarity. Right: 62% larger discriminative margin.

Figure 6: Failure mode geometry. Each type occupies a distinct similarity region. Embedding limitations dominate.

Routing	Dress	Shirt	Toptee	Avg	VLM%
Always VLM	22.4	27.2	24.8	24.8	100
Hybrid ( $\theta{=}0.3$ )	21.6	24.6	25.0	23.7	74
Hybrid ( $\theta{=}0.4$ )	21.3	24.1	24.9	23.4	92
Always Direct	17.6	24.1	24.8	22.2	0

Table 5: Hybrid query routing. FashionIQ full val. VLM% = fraction routed to VLM. R@10 (%).

Component	Claude	Nova Pro	No VLM
VLM captioning	1,169	449	0
CLIP encoding	25	25	25
FAISS search	3	3	3
Total	1,197	477	28

Table 6: Per-query latency breakdown (ms).

Composed Image Retrieval

Composed image retrieval finds images that match a reference image together with a text instruction describing the desired change.

See all 65 papers

FIRE-CIR: Fine-grained Reasoning for Composed Fashion Image Retrieval
Apr 10, 2026François Gardères, Camille-Sovanneary Gauthier, Jean Ponce +1Composed Image Retrieval VLM Reasoning
Louis Vuitton · Inria, École normale supérieure, CNRS, PSL Research University · Courant Institute of Mathematical Sciences and Center for Data Science, New York University
Towards Vision-Free CIR: Attribute-Augmented Scoring and LLM-Based Reranking for Zero-Shot Composed Image Retrieval
Jul 14, 2026Ryotaro Shimada, Yu-Chieh Lin, Yuji Nozawa +3Composed Image Retrieval
The University of Tokyo · Kioxia Corporation
Less is More: Concise Retrieval and Critic-based Re-ranking for Composed Image Retrieval
Oct 20, 2026Yuqian Zheng, Hyoju Ahn, Hyeongjin Ahn +1Composed Image Retrieval Image Retrieval
Department of Applied Artificial Intelligence, Sungkyunkwan University, Seoul, Republic of Korea · Department of Immersive Media Engineering, Sungkyunkwan University, Seoul, Republic of Korea · Jaume I University, Castellon, Spain

Apr 10, 2026J/K move · Enter open · S save

FIRE-CIR: Fine-grained Reasoning for Composed Fashion Image Retrieval

François Gardères, Camille-Sovanneary Gauthier, Jean Ponce, Shizhe Chen

Louis Vuitton · Inria, École normale supérieure, CNRS, PSL Research University · Courant Institute of Mathematical Sciences and Center for Data Science, New York University

Composed Image Retrieval VLM Reasoning

Open paper AskPDF arXiv

Composed image retrieval (CIR) aims to retrieve a target image that depicts a reference image modified by a textual description. While recent vision-language models (VLMs) achieve promising CIR performance by embedding images and text into a shared space for retrieval, they often fail to reason about what to preserve and what to change. This limitation hinders interpretability and yields suboptimal results, particularly in fine-grained domains like fashion. In this paper, we introduce FIRE-CIR, a model that brings compositional reasoning and interpretability to fashion CIR. Instead of relying solely on embedding similarity, FIRE-CIR performs question-driven visual reasoning: it automatically generates attribute-focused visual questions derived from the modification text, and verifies the corresponding visual evidence in both reference and candidate images. To train such a reasoning system, we automatically construct a large-scale fashion-specific visual question answering dataset, containing questions requiring either single- or dual-image analysis. During retrieval, our model leverages this explicit reasoning to re-rank candidate results, filtering out images inconsistent with the intended modifications. Experimental results on the Fashion IQ benchmark show that FIRE-CIR outperforms state-of-the-art methods in retrieval accuracy. It also provides interpretable, attribute-level insights into retrieval decisions.

Abstract

Composed Image Retrieval

Explore similar work

FIRE-CIR: Fine-grained Reasoning for Composed Fashion Image Retrieval

Towards Vision-Free CIR: Attribute-Augmented Scoring and LLM-Based Reranking for Zero-Shot Composed Image Retrieval

Less is More: Concise Retrieval and Critic-based Re-ranking for Composed Image Retrieval