Sketches offer designers a concise yet expressive medium for early-stage fashion ideation by specifying structure, silhouette, and spatial relationships, while textual descriptions complement sketches to convey material, color, and stylistic details. Effectively combining textual and visual modalities requires adherence to the sketch visual structure when leveraging the guidance of localized attributes from text. We present LOcalized Text and Sketch with multi-level guidance (LOTS), a framework that enhances fashion image generation by combining global sketch guidance with multiple localized sketch-text pairs. LOTS employs a Multi-level Conditioning Stage to independently encode local features within a shared latent space while maintaining global structural coordination. Then, the Diffusion Pair Guidance stage integrates both local and global conditioning via attention-based guidance within the diffusion model's multi-step denoising process. To validate our method, we develop Sketchy, the first fashion dataset where multiple text-sketch pairs are provided per image. Sketchy provides high-quality, clean sketches with a professional look and consistent structure. To assess robustness beyond this setting, we also include an "in the wild" split with non-expert sketches, featuring higher variability and imperfections. Experiments demonstrate that our method strengthens global structural adherence while leveraging richer localized semantic guidance, achieving improvement over state-of-the-art. The dataset, platform, and code are publicly available.
Figures & tables
Fig. 1: LOTS enables automation of the fashion design process at a new level of detail. The figure illustrates a design scenario where sketches are complemented by natural language descriptions to characterize garment material, style, and structure. LOTS represents a paradigm shift in design methodologies, advancing from global ( ) text with global sketch (IP-Adapter [ 8 ] ) and global text with localized ( ) sketches (Multi-ControlNet [ 6 ] ). Our approach adds localized sketch-text specifications (the coloured boxes), enabling fine-grained control over the layout and attributes of multiple garment items. All textual descriptions are shown in a contracted form for readability, see text.
Fig. 2: LOTS pipeline. 1. The first Multi-level conditioning stage constructs a conditioning representation spanning both local and global levels. Locally, the Modularized Pair-Centric Representation module (Sec. III-B ) handles each sketch–text pair independently: modality-specific, frozen encoders first map sketches and texts into their respective embeddings, which are then fused in the Pair-Former by integrating textual semantics with the spatial structure of the corresponding sketch. In parallel, the Global Conditioning (Sec. III-B ) derives a global representation from the full sketch and injects it via cross-attention to promote consistency and interaction across multiple pairs. 2. In the subsequent Diffusion Pair Guidance stage (Sec. III-C ), the multi-level embeddings are progressively incorporated into the diffusion process, together with the Global Context Description which drives the background generation and shapes the overall style. Rather than explicitly merging all pair representations upfront, conditioning is applied throughout the denoising process, enabling gradual integration and preventing the attribute leakage typically induced by early representation fusion.
Fig. 3: Overview of Sketchy. We build a hierarchical structure by pairing the garment part annotations to their related whole-body garment. Then, garment-level sketches and natural language descriptions are added based on off-the-shelf models and the in-the-wild sketch collection pipeline. The bar charts illustrate the frequency of the prevalent categories within the dataset, representing the total count of annotations where whole-body items (left) and garment parts (right) appear.
Fig. 4: Examples of sketches in the Sketchy and Sketchy in the Wild dataset. The left column presents automatically annotated sketches in Sketchy. The middle column shows collected human-drawn sketches in the Sketchy in the Wild subset. The right column displays the corresponding original fashion images. Human-drawn sketches exhibit higher subjective abstraction and stylistic variability.
Model
Conditioning
Global Quality
Compositional Alignment
Visual/Textual
FID ( ↓ )
GlobalCLIP ( ↑ )
LocalCLIP ( ↑ )
VQAScore ( ↑ )
L-VQAScore ( ↑ )
SSIM ( ↑ )
SD [ 4 ]
-/G
1.03
.613
.753
.694
.430
.652
SDXL [ 5 ]
-/G
1.09
.567
.757
.781
.547
.661
GLIGEN [ 31 ]
-/L
1.10
.544
.709
.282
.213
.594
ControlNet [ 6 ]
G/G
0.96
.633
.801
.709
.579
.623
Multi-ControlNet [ 6 ]
L/G
0.98
.624
.786
.687
.528
.638
TABLE I: Comparisons between LOTS and state-of-the-art sketch-to-image approaches on Sketchy. In the Conditioning column, L and G indicate whether the model accepts Local or Global inputs as Visual or Textual conditioning. We divide the table into three sections: Ay zero-shot approaches, Ay fine-tuned approaches, our prior and current proposed method Ay LOTS* and Ay LOTS . We highlight the best performance in bold and underline the second best .
Model
Conditioning
Global Quality
Compositional Alignment
Visual/Textual
FID ( ↓ )
GlobalCLIP ( ↑ )
LocalCLIP ( ↑ )
VQAScore ( ↑ )
L-VQAScore ( ↑ )
SSIM ( ↑ )
SD [ 4 ]
-/G
1.46
.614
.759
.703
.438
.645
SDXL [ 5 ]
-/G
1.46
.566
.759
.787
.541
.661
GLIGEN [ 31 ]
-/L
1.57
.538
.709
.293
.223
.599
ControlNet [ 6 ]
G/G
1.37
.620
.786
.710
.540
.616
Multi-ControlNet [ 6 ]
L/G
1.37
.619
.782
.683
.513
.635
TABLE II: Comparisons between LOTS and state-of-the-art sketch-to-image approaches on Sketchy in the wild split. In the Conditioning column, L and G indicate whether the model accepts Local or Global inputs as Visual or Textual conditioning. We divide the table into three sections: Ay zero-shot approaches, Ay fine-tuned approaches, our prior and current proposed method Ay LOTS* and Ay LOTS . We highlight the best performance in bold and underline the second best .
Fig. 5: Qualitative comparison of LOTS with our prior work LOTS* [ 12 ] , ControlNet [ 6 ] , IP-Adapter [ 8 ] , and Multi-T2I-adapter [ 7 ] , all in their fine-tuned versions. Given localized sketch-text pairs as conditioning inputs, LOTS better capture fine-grained attributes within the intended local regions of the generated images, effectively mitigating attribute confusion while maintaining strong global structural alignment.
Fig. 6: Qualitative comparison of LOTS with IP-Adapter [ 8 ] , and Multi-T2I-adapter [ 7 ] , all in their fine-tuned versions. Cropped views illustrate the details.
Fig. 7: Effects of different global context descriptions on the generation of LOTS. The text labels (top and bottom) indicate the specific global context prompts used in each generation. By changing it, we are able to customize general aspects such as the background and style of the model and the outfit.
Component
Choice
Global Quality
Compositional Alignment
FID ( ↓ )
GlobalCLIP ( ↑ )
LocalCLIP ( ↑ )
VQAScore ( ↑ )
L-VQAScore ( ↑ )
SSIM ( ↑ )
Global Conditioning
LOTS* (Ours)
0.79
.651
.818
.709
.692
.651
CONCAT
0.79
.651
.817
.769
.703
.644
ATTN
0.82
.640
.823
.700
.698
.684
Pair-Former
64-TOKEN
0.70
.656
.826
.748
.689
.676
Diffusion Guidance
POOL
0.74
.654
.823
.680
.612
.687
TABLE III: Ablation study comparing the impact of global conditioning, pair-former pooling and diffusion guidance on key metrics. We highlight the best performance in bold .
Attribute Localization
Structural Alignment
Model
Precision ( ↑ )
Recall ( ↑ )
F1 ( ↑ )
Preference Rate % ( ↑ )
SDXL [ 5 ]
.636
.754
.690
14.7
IP-Adapter [ 8 ]
.625
.139
.227
0.00
ControlNet [ 6 ]
.667
.516
.582
67.6
IP-Adapter [ 8 ]
.559
.384
.455
4.00
T2I-Adapter [ 7 ]
.463
.397
.427
3.00
TABLE IV: Results of subjective user studies on attribute localization and structural alignment between LOTS and selected top-performing models. For the attribute localization study, we highlight the best results in bold and underline the second best .
Fashion sketching is a cornerstone of design workflows, allowing rapid visualization of creative concepts prior to physical prototyping. Yet, progress in sketch-based fashion image synthesis has been hindered by the absence of large-scale, high-quality paired resources. To bridge this gap, we present GarmentSketch, a novel dataset comprising 26,249 fashion sketches across 21 garment categories, each paired with detailed textual descriptions. Captions were produced through a multi-stage pipeline that integrates multiple multimodal large language models (MLLMs) with human-in-the-loop refinement, ensuring both semantic accuracy and descriptive richness. We benchmark GarmentSketch on state-of-the-art generative models, providing baseline performance for sketch-guided text-to-image generation. Our experiments reveal both the promise and the current limitations of existing methods. By offering a comprehensive and richly annotated resource, GarmentSketch establishes a foundation for advancing sketch understanding, fine-grained fashion image generation, and creative human-AI collaboration in design. The dataset will be available at: https://khangbdd.github.io/garmentsketch.
Duong-Duy-Khang Bui, Minh-Tan Pham, Tam V. Nguyen +2
University of Science, Ho Chi Minh, Vietnam · Vietnam National University, Ho Chi Minh, Vietnam · University of Dayton, Ohio, United States
Recent research work on fashion outfit generation focuses on promoting visual consistency of garments by leveraging key information from reference image and text prompt. However, the potential of outfit generation remains underexplored, requiring comprehensive e-commercial dataset and elaborative utilization of multi-modal condition. In this paper, we propose a brand-new e-commerce dataset, named Fashion130k, with various occasions, models, and garment types. For the consistent generation of garment, we design a framework with Unified Multi-modal Condition (UMC) to align and integrate the text and visual prompts into generation model. Specifically, we explore an embedding refiner to extract the unified embeddings of multi-modal prompts, within which a Fusion Transformer is proposed to align the multi-modal embeddings by adjusting the modality gap between text and image. Based on unified embeddings, the attention in generation model is redesigned to emphasis the correlations between prompts and noise image, inducing that the noise image can select the pivotal tokens of prompts for consistent outfit generation. Our dataset and proposed framework offer a general and nuanced exploration of multi-modal prompts for generation models. Extensive experiments on real-world applications and benchmark demonstrate the effectiveness of UMC in visual consistency, achieving promising result than that of SoTA methods.
Sketching is one of the most natural ways for people to express visual ideas. Unlike the text prompts that most generative systems rely on today, a sketch captures spatial structure and intent directly, including where a designer wants highlights and shadows to appear. Yet current sketch-to-3D pipelines have no input for that intent: they bake the drawn shading into the texture as dark paint, and the light cannot be moved. We propose a system that transforms a rough garment sketch, marked with sparse annotations (a star for a highlight and a moon for a shadow), and a short text prompt describing the texture, into a textured 3D mesh with an explicit directional light, as a scene that can be relit and edited after generation. The two symbols are the most frequent proposals in an elicitation study with art students. From the marked sketch the system predicts the scene's directional light and, separately, a displacement field that bends the surface so that the moon-marked regions fall into shadow under that light, rather than being painted dark in the texture.
Ahmet Neçirvan Doğan, Eren Acar Başaran, Metin Sezgin +1
Koç University, Türkiye · University of Manchester, UK