Organizations: Department of Natural Language Processing Mohamed Bin Zayed University of Artificial Intelligence · Department of Computer Science and Engineering Indian Institute of Technology Jodhpur
Fashion Image Captioning (FIC) plays a vital role in enhancing user experience and product search in e-commerce platforms. Unlike natural scene image captioning, FIC requires fine-grained visual reasoning and knowledge of domain-specific terminology to capture subtle attributes such as neckline and closure types, graphic patterns, and dress silhouettes. Moreover, as fashion inventories evolve rapidly with new trends, styles, and frequently emerging vocabulary, developing training-free captioning solution becomes essential for scalability and real-world adaptability. Instruction-tuned vision-language models (VLMs) offer a promising solution to fashion image captioning dueto their strong zero-shot capabilities and natural language fluency. However, these general-purpose models often lack attribute-level coverage and precision, and tend to hallucinate or misidentify fine-grained fashion details, making them less suitable for high-fidelity applications like product cataloging or personalized recommendations. To address this, we propose RA-CoA (Retrieval-Augmented Chain-of-Attributes), a novel, training-free framework that disentangles fashion image captioning into two interpretable stages: (i) retrieval of relevant attribute sets from a product knowledge base, and (ii) attribute-level reasoning to generate the final caption. RA-CoA is a model-agnostic approach that works with frozen VLMs to improve fine-grained attribute precision in product captions without the need for fine-tuning. Extensive evaluations across diverse VLM model families under different prompting paradigms demonstrate that RA-CoA significantly improves caption quality, achieving an average gain of 26.3% METEOR score over zero-shot captioning. We make our code publicly available.
Figures & tables
Figure 1: Task Overview: Given a fashion image and the product name, existing training-free prompting paradigms may fail to provide attribute-rich, coherent, hallucination-free captions. Captioning with (a) Zero-shot: Produces only a generic description, missing key fine-grained attributes. (b) In-context learning Dong et al. (2022) : Improves coherence via exemplars but tends to copy attributes. (c) Chain-of-Thought Wu et al. (2025) : Performs stepwise reasoning yet fails in capturing all fine-grained attributes correctly. (d) RA-CoA (Ours): A retrieval-augmented chain-of-attributes framework that aggregates key attributes from similar catalog items and sequentially predicts their values. The predicted attributes, combined with ICL exemplars, guide caption generation toward coherent and attribute-faithful captions.
Figure 2: Overview of RA-CoA . Ⓐ Retrieval-augmented Chain-of-Attributes - For a query fashion image, we crop the target product region using Florence-2, retrieve top-K similar products from ProductKB, and aggregate unique attributes. Each attribute is paired with the image and product name and fed to VLM for value prediction. Ⓑ Caption Generation - The resulting attribute-value pairs and in-context exemplars (retrieved products) enable the VLM to generate fine-grained coherent captions. ⓒ ProductKB Construction - We construct our ProductKB by extracting structured attribute-value pairs from FashionGen captions using LLaMA-3.2-V-11B.
Model (# params)
Paradigm
BLEU-1
Avg. BLEU
Rouge-1
Rouge-2
Rouge-L
METEOR
LLM-Judge
TinyLLaVA (3B)
Zero-shot
12.3
3.5
17.1
2.4
11.4
12.2
17.2
CoT-i
11.0
3.3
18.6
2.7
12.6
11.0
10.1
CoT-e
7.8
2.3
12.8
1.8
8.7
8.6
8.1
ICL
13.1
4.8
24.5
4.9
18.7
13.7
36.5
ICL+CoT-e
2.9
1.2
9.2
2.3
7.5
7.9
15.2
RA-CoA
22.1
11.50
32.5
10.8
25.2
24.6
50.9
Table 1: Comparison of RA-CoA against different prompting paradigms across VLMs of varying scales. Δ (gray rows) represents the gain of RA-CoA over Zero-shot.
Figure 3: A selection of RA-CoA ’s results. Our proposed approach RA-CoA generates coherent captions with minimal hallucinations that align well with the human-annotated ground truth descriptions.
Model (# params)
Top-K
BLEU-1
Avg. BLEU
Rouge-1
Rouge-2
Rouge-L
METEOR
TinyLLaVA (3B)
1
22.1
11.50
32.5
10.8
25.2
24.6
2
20.0
9.8
30.5
9.4
23.2
22.1
3
18.7
8.9
29.2
8.6
21.9
21.0
Qwen-2.5VL (3B)
1
26.8
14.5
35.3
11.7
28.3
28.6
2
31.3
16.6
37.7
12.6
29.3
31.3
3
25.6
13.1
33.1
9.8
25.1
28.3
Table 2: Ablation study to quantify the impact of top-K retrievals for CoA in our proposed RA-CoA method.
Model (# params)
Method variant
BLEU-1
Avg.BLEU
Rouge-1
Rouge-2
Rouge-L
METEOR
TinyLLaVA (3B)
RA-CoA
22.1
11.50
32.5
10.8
25.2
24.6
RA-CoA w/o ICL exemplars
16.8
5.3
19.8
2.5
12.8
17.9
RA-CoA with Oracle Attributes
22.9
11.2
34.1
10.5
26.4
24.7
RA-CoA with Oracle Attribute-Values
53.9
44.9
74.0
60.4
71.3
71.6
Qwen-2.5VL (3B)
RA-CoA
26.8
14.5
35.3
11.7
28.3
28.6
RA-CoA w/o ICL exemplars
17.7
5.7
22.6
2.8
14.2
19.8
Table 3: Ablation study to (1) quantify the importance of retrieval-augmented ICL exemplars, and (2) quantify the gap of RA-CoA with respect to oracle variants.
Figure 4: Comparison of CLIP-only and Florence-CLIP for retrieval in RA-CoA (Section 3.2.1 ).
Paradigm
Input image
BLEU-1
Avg.BLEU
Rouge-1
Rouge-2
Rouge-L
METEOR
Zeroshot
Cropped Product
9.6
2.7
19.1
2.9
12.6
11.7
CoT-i
13.0
3.8
18.0
2.9
12.6
15.0
CoT-e
7.0
2.1
16.8
2.4
12.4
13.2
ICL
29.8
17.3
35
13.3
28.7
31.6
RA-CoA
39.4
24.3
41.8
18.0
36.7
38.8
RA-CoA
Full Image
38.6
23.7
41.0
17.4
36.0
38.1
Table 4: Comparison of using full image vs. Florence-2–cropped product region as VLM input for captioning. Cropping offers a modest gain while the full image already yields strong results. All results are reported for InternVL2 (8B) as base VLM.
ProductKB size
BLEU-1
Avg.BLEU
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
10K
36.2
21.1
38.8
15.1
33.4
35.5
20K
37.6
22.7
40.3
16.6
35.2
37.1
40K
38.5
23.5
41.2
17.2
35.9
37.9
60K (Full)
38.1
23.7
41.0
17.4
36.0
38.1
Table 5: Effect of ProductKB size on captioning performance using InternVL2-8B within the RA-CoA framework. Performance improves with KB size, but gains saturate beyond 20K entries, indicating that RA-CoA remains data-efficient even with smaller knowledge bases.
% corrupted
BLEU-1
Avg.BLEU
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
0%
38.1
23.7
41.0
17.4
36.0
38.1
25%
37.0
21.8
39.5
16.1
34.5
36.5
50%
36.9
21.3
39.2
15.3
33.9
35.9
75%
35.7
19.9
37.7
13.8
32.4
34.3
Table 6: Effect of noisy ProductKB entries on captioning performance using InternVL2-8B within the RA-CoA framework. Performance decreases with increasing noise in attribute values, suggesting that RA-CoA is relatively robust to corrupted ProductKB entries.
% key-value pairs dropped
BLEU-1
Avg.BLEU
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
0%
38.1
23.7
41.0
17.4
36.0
38.1
25%
34.4
20.5
39.2
16.4
34.3
33.9
50%
28.4
17.1
37.6
16.4
33.1
29.7
75%
21.1
13.1
35.9
16.7
31.6
25.5
Table 7: Effect of sparse ProductKB entries on captioning performance using InternVL2-8B within the RA-CoA framework. Performance degrades as sparsity in the ProductKB increases, as it affects retrieval quality and attribute guidance.
Method
Fine-tuning
BLEU-1
Avg.BLEU
Rouge-1
Rouge-2
ROUGE-L
METEOR
UniFashion
✓
11.4
3.9
16.5
4.1
12.5
16.2
RA-CoA (Ours)
✗
22.0
7.7
26.1
5.7
18.3
20.2
Table 8: Quantitative comparison of RA-CoA with UniFashion Zhao et al. (2024) (prior SOTA supervised method) showcasing real-world generalization of RA-CoA despite being completely training-free.
Method
BLEU-1
Avg.BLEU
Rouge-1
Rouge-2
ROUGE-L
METEOR
LLM-Judge
Zero-shot
4.0
1.4
22.8
2.0
17.4
9.2
65.7
CoT-i
38.6
17.7
36.5
11.2
28.3
27.4
75.1
ICL
71.0
55.0
68.2
44.8
67.5
72.8
34.2
RA-CoA (Ours)
74.5
60.6
74.8
51.3
71.4
73.2
77.2
Table 9: Quantitative comparison of RA-CoA with other VLM paradigms on the Furniture Dataset, showcasing generalization of our approach beyond fashion domain.
Method
Preference (%)
Zero-shot
3.7
CoT-i (Implicit CoT)
6.7
CoT-e (Explicit CoT)
7.0
In-Context Learning (ICL)
14.0
RA-CoA (Ours)
68.6
Table 10: User preference study over 300 unique test images using InternVL2-8B as the backbone. RA-CoA is preferred in 68.6% of cases, reflecting higher perceived coherence, completeness, and usefulness.
Figure 5: Error analysis into RA-CoA ’s generations. The text in red denotes incorrect/hallucinated features. The text in blue in the GT caption indicates features missed in the generated caption.
Step
Time (seconds)
ROI crop using Florence-2
0.30
CLIP embedding extraction for ROI
0.04
Retrieval from ProductKB (Section 3.2.1 )
0.01
Chain-of-attributes (Section 3.2 )
1.40
Attribute-value aware captioning (Section 3.2.4 )
1.80
Average inference time per sample (Total)
3.5
Table 11: Computational latency analysis of RA-CoA ’s components.
Composed image retrieval retrieves a target image using a composed query of a reference image and a modified text description. In the fashion domain, this task requires understanding subtle attribute variations such as color, pattern, and texture. However, existing approaches face limitations due to scarce annotated data and simplistic negative sampling. We propose a novel framework that integrates a multi-modal large language model (LLaVA) to generate attribute-aware triplets and introduces a two-stage fine-tuning strategy to enhance contrastive learning. We leverage pretrained vision-language models, such as CLIP-ViT/B32, to generate and concatenate sentence-level prompts with the relative caption and to scale the number of negatives using static representations. Experimental results demonstrate enhanced compositional reasoning and improved fine-grained retrieval behavior, underscoring the feasibility and potential of the proposed framework for fashion retrieval.
Nguyen Cao Hoang, Hoang Bui Le, Nam Vo Hoang +1
University of Science, VNU-HCM, Ho Chi Minh, Vietnam · Vietnam National University, Ho Chi Minh, Vietnam
Composed Image Retrieval (CIR) retrieves products matching a reference image modified by natural language---a core capability for eCommerce search. Existing training-free methods fuse CLIP embeddings via arithmetic, but we identify a fundamental limitation: user text modifications are attribute-sparse---a user who types make it red with shorter sleeves'' specifies what to change but not what to keep (the floral pattern, the V-neck, the fit-and-flare silhouette). We introduce language-space fusion: a Vision-Language Model (VLM) jointly interprets the reference image and modification to generate a visually grounded product-title caption---e.g., Red floral fit-and-flare dress with short sleeves and V-neck''---that fills in the missing attributes from the image. Our method, CaptionFuse, requires no task-specific training and is model-agnostic. A controlled experiment confirms that attribute content--- not linguistic style---drives the gains: imperative and descriptive captions with identical attributes achieve equal R@10, while both outperform raw modifications by +6.5 points.
Zhao Gao, Mina Metias, Mohamed Noordeen Alaudeen +1
We present Fashion Florence, a Florence-2 vision-language model fine-tuned with LoRA to extract structured fashion attributes from clothing images. Given a single photograph, the model generates a JSON object containing category, color, material, style tags, and occasion tags, structured output suitable for direct programmatic consumption by downstream recommendation and retrieval systems. Fine-tuning data is derived from the iMaterialist Fashion dataset (228 labels), where we collapse fine-grained annotations into a compact 6-category, 16-color, 19-style schema via rule-based label engineering. We apply LoRA (r=16, alpha=32) to all decoder linear layers, training for 3 epochs on 3,688 examples. On a held-out test set of 461 images, Fashion Florence achieves 94.6% category accuracy and 63.0% material accuracy, compared to 89.3% / 43.3% for GPT-4o-mini and 87.4% for Gemini 2.5 Flash. Fashion Florence produces valid JSON in 99.8% of outputs while running at 0.77B parameters on a single GPU at zero marginal inference cost. Style tag F1 reaches 0.753 vs. 0.612 (Gemini) and 0.398 (GPT-4o-mini). The model is deployed as a Hugging Face Space and integrated into Loom, an open-source outfit recommendation system.