Autonomous Fashion Outfit Composition via Unified Aesthetic Foresight Model

Kaicheng Pang; Xingxing Zou; Ruohan Xu; Waikeung Wong

cs.LGAug 14, 2026

Autonomous Fashion Outfit Composition via Unified Aesthetic Foresight Model

Authors: Kaicheng Pang, Xingxing Zou, Ruohan Xu, Waikeung Wong

Organizations: Laboratory for Artificial Intelligence in Design Hong Kong SAR · Hong Kong Polytechnic University Hong Kong SAR · The University of Queensland Brisbane, Australia

Abstract

Fashion Outfit Composition (FOC) requires sequentially assembling fashion items into a stylistically cohesive ensemble. Existing works struggle to model this step-by-step process effectively, primarily because they fail to jointly optimize the two critical capabilities required for FOC: intermediate outfit value evaluation and complementary item prediction. This structural disconnect, compounded by the severe sparsity of step-wise aesthetic signals, leaves them without a mechanism to autonomously determine when to stop the composition process. To address these challenges, we introduce the Unified Aesthetic Foresight Model (UAFM), which seamlessly unifies both capabilities into a dual-head architecture over a shared backbone. Crucially, we formulate FOC as a deterministic Markov Decision Process and optimize a value head via a post-decision state temporal difference (TD) objective to recursively backpropagate sparse terminal aesthetic rewards to intermediate states. This enables the model to accurately estimate the potential of a partial outfit evolving into a compatible ensemble. Consequently, UAFM can compute step-wise marginal aesthetic gains for dynamic termination. Furthermore, this unified design strictly aligns aesthetic evaluation and complementary item prediction within a shared representation manifold, intrinsically regularizing the combinatorial search space. Extensive experiments on the Polyvore-Outfits dataset demonstrate that UAFM establishes a new state-of-the-art across both FOC and conventional fashion tasks. Extensive ablation studies further confirm that our post-decision state TD formulation provides the aesthetic value prediction necessary for autonomous dynamic termination, while validating the synergistic benefits of our unified dual-head architecture.

Figures & tables

Figure 1: Fashion Outfit Composition: Challenge and Paradigm. Conventional paradigms face severe reward sparsity during sequential composition, as positive supervision is restricted to complete human-curated outfits, leaving intermediate partial states unmonitored (Panel 1). Addressing FOC requires dual capabilities: bootstrapping intermediate state values from sparse terminal rewards and predicting next-item intent. Unifying intermediate aesthetic evaluation (Value Head) and candidate selection (Policy Head) onto a shared backbone enables autonomous dynamic termination by tracking diminishing marginal aesthetic gains (Panel 2). Our method generates cohesive ensembles, whereas conventional baselines suffer from severe pattern conflicts and item overlaps (Panel 3).

Figure 2: Overview of the UAFM architecture and training pipeline. (A) MDP Trajectory: Human-curated outfits are decomposed into state-action trajectories, governed by step penalties and terminal aesthetic rewards. (B) UAFM Architecture: A shared encoder backbone jointly parameterizes the value head (aesthetic potential) and the policy head (stylistic intent) within a dual-head framework. (C) Joint Training Objective: (C1) Value Head Training employs a proposed post-decision state TD objective to recursively backpropagate sparse terminal rewards to early intermediate states; (C2) Policy Head Training aligns latent intents via InfoNCE-based contrastive learning.

Method	H. Pref.	VAS	Cate. Align	Cate. Dup. $\downarrow$	Vis. Red. $\downarrow$	Latency (s) $\downarrow$
Random	-	2.50 ± 0.04	42.34 ± 0.15	17.84 ± 0.13	0.18 ± 0.03	0.0078
Type-Aware	-	2.69 ± 0.04	41.15 ± 0.28	21.46 ± 0.11	1.39 ± 0.08	0.15
OutfitTrans.	25.75%	3.18 ± 0.03	62.21 ± 0.23	20.58 ± 0.25	6.10 ± 0.11	0.14
Gemini-3.5	28.08%	3.28 ± 0.03	64.93 ± 0.23	15.48 ± 0.17	8.60 ± 0.15	6.35
UAFM	46.17%	3.57 ± 0.02	72.18 ± 0.27	5.69 ± 0.15	1.20 ± 0.09	0.13

Table 1: Results of composed outfits quality across different baselines. Bold and underline indicate the best and second-best performances, respectively. Metrics with

\downarrow

denote lower values are better. Scores for VAS , Cate. Align (%) , Cate. Dup. (%) , and Vis. Red. (%) report 5-fold Mean

\pm

SE.

Figure 3: Radar chart of VLLM Aesthetic Scores (VAS). Composed outfits are evaluated across five aesthetic dimensions along with the overall average score.

Method	AUC	FITB (%)	R@10 (%)	R@30 (%)	R@50 (%)
Type-Aware	0.84	55.65	3.66	8.26	11.98
SCE-Net	-	53.67	4.41	9.85	13.87
CSA-Net	0.87	59.26	5.93	12.31	17.85
LLM-DPO	0.81	61.00	-	-	-
OutfitTransformer	0.90	69.01	10.51	20.33	27.64
UAFM (Ours)	0.93	70.84	12.05	22.63	29.50

Table 2: Performance comparison on downstream discriminative tasks on disjoint version of the Polyvore-Outfits dataset. Best results are highlighted in bold .

Figure 4: Ablation on training objectives. Part (a) compares average composed outfit length (blue bars) against GT outfit length (red dashed line) and natural stop rate (green bars). Part (b) evaluate predicted aesthetic scores across composition steps with 3-fold cross-validation error bars.

Architecture	Composed Outfit Quality				Discriminative Performance
Architecture	VAS	Cate. Align %	Cate. Dup. % $\downarrow$	Vis. Red. % $\downarrow$	AUC	FITB	R@10	R@30	R@50
BCE	3.33 ± 0.02	55.23 ± 0.27	15.89 ± 0.11	3.70 ± 0.07	0.96	70.41	11.77	20.10	29.03
Separate	3.27 ± 0.02	67.73 ± 0.29	18.01 ± 0.32	6.75 ± 0.22	0.84	68.57	10.09	19.72	26.25
UAFM	3.60 ± 0.02	78.20 ± 0.49	1.29 ± 0.07	0.34 ± 0.04	0.93	70.84	12.05	22.63	29.50

Table 3: Performance comparison among the BCE variant, the Separate variant, and UAFM . All models share the identical backbone and training recipe, operating under dynamic termination without explicit length constraints. Best results are highlighted in bold .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Strategy	Latency	Cate. Align % $\uparrow$	Cate. Dup. % $\downarrow$	Vis. Red. % $\downarrow$	VLLM Aesthetic Score $\uparrow$
Strategy	Latency	Cate. Align % $\uparrow$	Cate. Dup. % $\downarrow$	Vis. Red. % $\downarrow$	Col.	Bal.	Sty.	Tex.	Wear.	Overall
Value-Greedy	0.028s	78.99 ± 0.48	1.01 ± 0.07	0.37 ± 0.02	3.93	3.47	3.77	3.24	3.52	3.59 ± 0.02
Reranking	0.24s	78.80 ± 0.43	1.07 ± 0.06	0.39 ± 0.06	3.93	3.48	3.78	3.31	3.48	3.60 ± 0.03
Beam Search	0.13s	78.20 ± 0.49	1.29 ± 0.07	0.34 ± 0.04	3.94	3.50	3.77	3.30	3.50	3.60 ± 0.03
Policy-Only	0.021 s	53.86 ± 0.14	6.82 ± 0.21	2.78 ± 0.13	3.91	3.26	3.68	3.26	3.03	3.43 ± 0.03

Appendix

Table 4: Comparison of different decoding strategies utilizing UAFM for outfit composition.

Figure 5: Interface screenshot of the Human Preference A/B testing questionnaire on Microsoft Forms. Evaluators conduct blind pairwise comparisons between outfits composed by different algorithms under standardized visual criteria.

Figure 6: Qualitative comparison of composed outfits across different paradigms. Given an initial seed garment, each system sequentially retrieves complementary items to assemble a complete ensemble. Compared with baseline paradigms, UAFM (Ours) achieves superior stylistic cohesion, color harmony, and category diversity, closely resembling human compositions ( Ground Truth ).

Apr 10, 2026

FashionStylist: An Expert Knowledge-enhanced Multimodal Dataset for Fashion Understanding

Fashion understanding requires both visual perception and expert-level reasoning about style, occasion, compatibility, and outfit rationale. However, existing fashion datasets remain fragmented and task-specific, often focusing on item attributes, outfit co-occurrence, or weak textual supervision, and thus provide limited support for holistic outfit understanding. In this paper, we introduce FashionStylist, an expert-annotated benchmark for holistic and expert-level fashion understanding. Constructed through a dedicated fashion-expert annotation pipeline, FashionStylist provides professionally grounded annotations at both the item and outfit levels. It supports three representative tasks: outfit-to-item grounding, outfit completion, and outfit evaluation. These tasks cover realistic item recovery from complex outfits with layering and accessories, compatibility-aware composition beyond co-occurrence matching, and expert-level assessment of style, season, occasion, and overall coherence. Experimental results show that FashionStylist serves not only as a unified benchmark for multiple fashion tasks, but also as an effective training resource for improving grounding, completion, and outfit-level semantic evaluation in MLLM-based fashion systems.

Kaidong Feng, Zhuoxuan Huang, Huizhong Guo +7

Yanshan University Qinhuangdao, China · Central South University Changsha, China · Zhejiang University Hangzhou, China +3

Multimodal Understanding

Ask PDF

Mar 3, 2026

OmniFashion: Towards Generalist Fashion Intelligence via Multi-Task Vision-Language Learning

Fashion intelligence spans multiple tasks, i.e., retrieval, recommendation, recognition, and dialogue, yet remains hindered by fragmented supervision and incomplete fashion annotations. These limitations jointly restrict the formation of consistent visual-semantic structures, preventing recent vision-language models (VLMs) from serving as a generalist fashion brain that unifies understanding and reasoning across tasks. Therefore, we construct FashionX, a million-scale dataset that exhaustively annotates visible fashion items within an outfit and organizes attributes from global to part-level. Built upon this foundation, we propose OmniFashion, a unified vision-language framework that bridges diverse fashion tasks under a unified fashion dialogue paradigm, enabling both multi-task reasoning and interactive dialogue. Experiments on multi-subtasks and retrieval benchmarks show that OmniFashion achieves strong task-level accuracy and cross-task generalization, highlighting its offering of a scalable path toward universal, dialogue-oriented fashion intelligence.

Zhengwei Yang, Andi Long, Hao Li +3

National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University · Harbin Institute of Technology

Vision-Language Models Large Vision-Language Models Image Retrieval Recommender Systems

Ask PDF

Apr 8, 2026

VersaVogue: Visual Expert Orchestration and Preference Alignment for Unified Fashion Synthesis

Diffusion models have driven remarkable advancements in fashion image generation, yet prior works usually treat garment generation and virtual dressing as separate problems, limiting their flexibility in real-world fashion workflows. Moreover, fashion image synthesis under multi-source heterogeneous conditions remains challenging, as existing methods typically rely on simple feature concatenation or static layer-wise injection, which often causes attribute entanglement and semantic interference. To address these issues, we propose VersaVogue, a unified framework for multi-condition controllable fashion synthesis that jointly supports garment generation and virtual dressing, corresponding to the design and showcase stages of the fashion lifecycle. Specifically, we introduce a trait-routing attention (TA) module that leverages a mixture-of-experts mechanism to dynamically route condition features to the most compatible experts and generative layers, enabling disentangled injection of visual attributes such as texture, shape, and color. To further improve realism and controllability, we develop an automated multi-perspective preference optimization (MPO) pipeline that constructs preference data without human annotation or task-specific reward models. By combining evaluators of content fidelity, textual alignment, and perceptual quality, MPO identifies reliable preference pairs, which are then used to optimize the model via direct preference optimization (DPO). Extensive experiments on both garment generation and virtual dressing benchmarks demonstrate that VersaVogue consistently outperforms existing methods in visual fidelity, semantic consistency, and fine-grained controllability.

Jian Yu, Fei Shen, Cong Wang +4

Nanjing University of Science and Technology · National University of Singapore · Nanjing University +1

Image Generation Controllable Image Generation Virtual Try-On

Ask PDF

Abstract

Explore similar work

FashionStylist: An Expert Knowledge-enhanced Multimodal Dataset for Fashion Understanding

OmniFashion: Towards Generalist Fashion Intelligence via Multi-Task Vision-Language Learning

VersaVogue: Visual Expert Orchestration and Preference Alignment for Unified Fashion Synthesis