We focus on the task of retrieving nail design images based on dense intent descriptions, which represent multi-layered user intent for nail designs. This is challenging because such descriptions specify unconstrained painted elements and pre-manufactured embellishments as well as visual characteristics, themes, and overall impressions. In addition to these descriptions, we assume that users provide palette queries by specifying zero or more colors via a color picker, enabling the expression of subtle and continuous color nuances. Existing vision-language foundation models often struggle to incorporate such descriptions and palettes. To address this, we propose NaiLIA, a multimodal retrieval method for nail design images, which comprehensively aligns with dense intent descriptions and palette queries during retrieval. Our approach introduces a relaxed loss based on confidence scores for unlabeled images that can align with the descriptions. To evaluate NaiLIA, we constructed a benchmark consisting of 10,625 images collected from people with diverse cultural backgrounds. The images were annotated with long and dense intent descriptions given by over 200 annotators. Experimental results demonstrate that NaiLIA outperforms standard methods.
Figures & tables
Figure 1: A typical use case for our task. The user inputs a dense intent description and an optional palette query. The palette query allows the user to select zero or more colors through a color picker interface. The model should rank the leftmost image higher than the middle image (painted in a darker purple than specified by the palette query) and the rightmost image (featuring realistic shell ornaments rather than the intended shell-inspired design). Details of this task are provided in Section 3.1 .
Figure 2: Architecture of NaiLIA. A language-palette representation l+ is extracted from xtxt and xpal by the Intent-Palette Fusion Module, and a visual representation v is extracted from ximg by the Visual Design Fusion Module. In the Confidence-based Relaxed Alignment Module, unlabeled positives are assigned, and the loss is calculated considering the unlabeled positives. Here, Enc. denotes the encoder.
Figure 3: Examples of cij estimated by the MLLM for the following xtxt : “Please put long fake nails on my nails and make them pink only at the base and the rest should be a fancy design with strawberries.” Positives, unlabeled positives, and negatives are framed in green, yellow, and red, respectively.
Method
w/ xpal
NAIL-STAR
Marqo Fashion200K
MRR↑ [%]
R@1↑ [%]
R@10↑ [%]
MRR↑ [%]
R@1↑ [%]
R@10↑ [%]
CLIP [ 46 ]
N
25.2 ± 0.1
15.5 ± 0.3
44.4 ± 0.3
61.7 ± 0.5
47.6 ± 0.7
88.8 ± 0.1
FashionViL [ 12 ]
N
20.8 ± 0.6
11.1 ± 0.9
40.9 ± 1.4
61.7 ± 0.5
47.3 ± 0.5
89.1 ± 0.5
FAME-ViL [ 13 ]
N
45.1 ± 0.3
31.7 ± 0.7
71.8 ± 0.3
65.3 ± 0.5
51.9 ± 0.8
90.5 ± 0.3
BEiT-3 [ 57 ]
N
53.9 ± 0.4
40.6 ± 0.6
79.8 ± 0.2
66.2 ± 0.8
52.8 ± 0.9
90.9 ± 0.5
BLIP-2 [ 27 ]
N
33.3 ± 1.0
20.8 ± 0.9
58.7 ± 1.2
75.3 ± 0.1
65.2 ± 0.1
94.9 ± 0.1
Table 1: Quantitative comparison of NaiLIA and baseline methods on the NAIL-STAR and Marqo Fashion200K [ 38 ] benchmarks. All models were fine-tuned on their respective training sets. Here, w/ xpal indicates whether the model takes palette queries as input. The scores present the average and standard deviation of five trials. The highest score for each metric is in bold , and the second-highest score is underlined . The results in the zero-shot setting are presented in Appendix H.1 .
Figure 4: Qualitative results of the proposed method (NaiLIA) and a baseline method (SigLIP) [ 67 ] for the following xtxt and xpal : (a) “I’d like my nails to have a cute, teenage vibe. I’d love a pink base with floral patterns and maybe some character accessories. Can we do a long nail shape?” with (#ffd3e5). (b) “I would like a fairy-tale nail design based on purple and pink.” with (#fff4f4) and (#bfc5ff). (c) “I’d like a colorful and flashy nail design. Please add a large flower nail stone to the ring finger. The tips of the nail tips should be square-shaped.” in the setting without xpal . (d) “red dress with a high waist. the dress is made of a stretchy material and has a flowy skirt. the dress is a formal style and is suitable for a special occasion.” with (#ff1f35). The top-5 retrieved images are shown. Positives and unlabeled positives are enclosed in green and yellow frames, respectively. Examples (a), (b), and (c) show successful cases of NaiLIA on the NAIL-STAR benchmark, whereas (d) is a successful case on the Marqo Fashion200K benchmark. Additional qualitative results, including cases where xpal consists of more than three colors, are presented in Appendix K .
Model
IPFM
VDFM
CRAM
MRR↑ [%]
R@1↑ [%]
R@10↑ [%]
lMDD
lNNP
vs
va
vn
loss
(a)
✓
✓
✓
✓
✓
CRC
67.6 ± 0.5
56.4 ± 0.7
88.0 ± 0.3
(b)
✓
✓
✓
✓
CRC
66.0 ± 0.5
54.9 ± 0.7
87.1 ± 0.2
(c)
✓
✓
✓
✓
CRC
65.6 ± 0.8
54.5 ± 0.9
87.2 ± 0.6
(d)
✓
✓
✓
CRC
63.0 ± 0.5
51.6 ± 0.6
84.8 ± 0.5
(e)
✓
✓
✓
✓
CRC
66.4 ± 0.4
55.1 ± 0.7
87.3 ± 0.3
Table 2: Ablation study on the core modules in our method. The average and standard deviation from five trials are reported. The highest score for each metric is in bold . The additional ablation study is presented in Appendix H.3 .
Model
MRR↑ [%]
R@1↑ [%]
R@10↑ [%]
CLIP [ 46 ] w/ InfoNCE
25.2 ± 0.1
15.5 ± 0.3
44.4 ± 0.3
CLIP [ 46 ] w/ CRC loss
25.9 ± 0.1
16.5 ± 0.3
44.8 ± 0.6
Table 3: Quantitative comparison of CLIP fine-tuned with InfoNCE vs. CRC loss on the NAIL-STAR benchmark.
Method
w/ xpal
MRR↑ [%]
R@1↑ [%]
R@10↑ [%]
CLIP [ 46 ]
N
25.2 ± 0.1
15.5 ± 0.3
44.4 ± 0.3
CLIP [ 46 ]
Y
29.7 ± 0.2
21.3 ± 0.3
48.7 ± 0.2
SigLIP [ 67 ]
N
58.8 ± 0.1
47.5 ± 0.1
82.1 ± 0.2
SigLIP [ 67 ]
Y
63.3 ± 0.1
53.4 ± 0.1
86.2 ± 0.1
Table 4: Quantitative results of applying palette queries to CLIP and SigLIP on the NAIL-STAR benchmark.
Figure 5: Examples of cij estimated by the MLLM in CRAM. Positive, unlabeled positive, and negative labels are framed in green, yellow, and red, respectively, with unlabeled positives identified based on cij .
Figure 6: Hue histogram of nail design images. We compute the distribution of hue values (HSV space, 0–360°) using all pixels within segmented nail regions in the NAIL-STAR benchmark. To visualize hue independent of saturation and value, the color of each bar corresponds to the bin center with fixed parameters ( S=1,V=1 ). The vertical axis indicates the pixel proportion relative to the total segmented area across the dataset.
Figure 7: Description annotation interface. The annotators were requested to input a dense intent description for the displayed nail design image into the text box below the image.
Figure 8: Palette query annotation interface. The annotators were requested to specify a palette query for the displayed nail design by using a color picker, selecting any number of colors (0–5).
Figure 9: The word frequency distribution of the NAIL-STAR benchmark. The frequency distributions of the 100 most common nouns, adjectives, and all words are presented. Singular and plural forms of nouns, comparative and superlative forms of adjectives, and present and past participles of verbs were counted independently. Stopwords were removed.
Method
MRR↑ [%]
R@1↑ [%]
R@10↑ [%]
CLIP [ 46 ]
12.4
6.5
23.7
FashionViL [ 12 ]
2.3
0.6
4.8
FAME-ViL [ 13 ]
20.2
11.2
39.3
BEiT-3 [ 57 ]
34.9
23.5
57.9
BLIP-2 [ 27 ]
14.4
7.5
28.0
SigLIP [ 67 ]
31.4
21.3
51.6
Table 5: Quantitative results of baseline methods on the NAIL-STAR benchmark in the zero-shot settings. The scores are reported based on a single trial.
Token Length
#
MRR↑ [%]
R@1↑ [%]
R@10↑ [%]
0–19
781
57.9
45.3
82.5
20–29
560
60.0
48.0
81.4
30–39
187
68.4
58.3
89.3
40–69
72
78.6
69.4
94.4
Total
1600
60.8
48.9
83.4
Table 6: Retrieval performance of NaiLIA across subsets grouped by description token length on the NAIL-STAR benchmark, evaluated in the description-only setting without xpal . The scores are reported based on a single trial.
Model
MRR↑ [%]
R@1↑ [%]
R@10↑ [%]
(a) NaiLIA (full)
82.5
74.5
96.6
(b) w/o vs
81.6
73.3
95.7
(c) w/o va
41.1
28.5
65.7
(d) w/o vn
78.8
69.7
95.1
Table 7: Ablation study of VDFM on the Marqo Fashion200K. The scores are reported based on a single trial.
Figure 10: Sensitivity analysis of the CRC loss with respect to the confidence threshold θ and the balance terms λUP and λN . For each curve, we varied one hyperparameter over {0.1,0.3,0.5,0.7,0.9} while keeping the other two fixed at their default values ( θ=0.5 , λUP=0.7 , λN=0.7 ). The scores are reported based on a single trial.
Errors Type
Description
#Error (desc-only)
#Error (full)
Comprehension
Comprehension errors in patterns and decorative elements
19
20
Comprehension errors in design themes
17
19
Comprehension errors in color-related expressions
13
6
Comprehension errors in nail shape and decoration placement
11
11
Comprehension errors in the correspondence between fingers and design
5
7
Ambiguity
Ambiguity in color description
18
9
Table 8: Number of failure cases in each category for NaiLIA in the description-only and full settings on the NAIL-STAR benchmark. The largest count in each setting is highlighted in bold .
Figure 11: Additional qualitative results of NaiLIA in the description-only setting, where only xtxt is provided. The top-5 retrieved images are shown. Positives and unlabeled positives are enclosed in green and yellow frames, respectively.
Figure 12: Additional qualitative results of NaiLIA in the setting where both xtxt and xpal are provided. The top-5 retrieved images are shown. Positives and unlabeled positives are enclosed in green and yellow frames, respectively.
Figure 13: Additional qualitative results of NaiLIA in the setting where both xtxt and xpal are provided. The top-5 retrieved images are shown. Positives and unlabeled positives are enclosed in green and yellow frames, respectively.
Modern retrieval systems must both be automated and interactive, allowing users to search and refine results in real time. We present AiSearch, a flexible multimodal retrieval framework that leverages the zero shot capabilities of Vision Language Models (VLMs) for natural language search over images and videos. AiSearch supports interactive search refinement through user feedback to tailor results to the user's intent, and allows visual benchmarking across multiple VLMs, enabling users to select the most suitable model for their task.
Ali Koksal, Mei Chee Leong, Vicky Sintunata +2
Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore
Cultural garments pose a unique challenge for visual retrieval systems, as their identity often depends on subtle structural and symbolic details that are poorly captured by standard AI models. We introduce VietFashion, a new benchmark for sketch-text composed image retrieval centered on the Ao Dai, a traditional Vietnamese garment. VietFashion enables designers and researchers to retrieve culturally meaningful outfits using a combination of hand-drawn sketches, which convey garment structure, and textual descriptions, which encode cultural semantics. The dataset is initialized with 650 sketches and expanded using generative models to produce over 21,000 photorealistic images with aligned captions. Textual prompts that describe detailed outfit attributes, which are extracted from fashion magazines to ensure authenticity and diversity. To better reflect the inherent ambiguity of design intent, VietFashion adopts a multi-target retrieval setting, where a single query may correspond to multiple valid results. We establish standardized evaluation protocols and benchmark state-of-the-art composed image retrieval methods. Experimental results reveal significant performance gaps in modeling fine-grained cultural semantics and multi-modal composition, positioning VietFashion as a challenging benchmark for fine-grained fashion retrieval. The dataset is publicly available at: https://hng0303.github.io/VietFashion.
Hoang-Nguyen Cao, Le-Hoang Bui, Dinh-Khoi Vo +2
University of Science · Ho Chi Minh City, Vietnam · Vietnam National University
Composed image retrieval retrieves a target image using a composed query of a reference image and a modified text description. In the fashion domain, this task requires understanding subtle attribute variations such as color, pattern, and texture. However, existing approaches face limitations due to scarce annotated data and simplistic negative sampling. We propose a novel framework that integrates a multi-modal large language model (LLaVA) to generate attribute-aware triplets and introduces a two-stage fine-tuning strategy to enhance contrastive learning. We leverage pretrained vision-language models, such as CLIP-ViT/B32, to generate and concatenate sentence-level prompts with the relative caption and to scale the number of negatives using static representations. Experimental results demonstrate enhanced compositional reasoning and improved fine-grained retrieval behavior, underscoring the feasibility and potential of the proposed framework for fashion retrieval.
Nguyen Cao Hoang, Hoang Bui Le, Nam Vo Hoang +1
University of Science, VNU-HCM, Ho Chi Minh, Vietnam · Vietnam National University, Ho Chi Minh, Vietnam