Autonomous Fashion Outfit Composition via Unified Aesthetic Foresight Model

Kaicheng Pang; Xingxing Zou; Ruohan Xu; Waikeung Wong

cs.LGAug 14, 2026

Autonomous Fashion Outfit Composition via Unified Aesthetic Foresight Model

Authors: Kaicheng Pang, Xingxing Zou, Ruohan Xu, Waikeung Wong

Organizations: Laboratory for Artificial Intelligence in Design Hong Kong SAR · Hong Kong Polytechnic University Hong Kong SAR · The University of Queensland Brisbane, Australia

Abstract

Fashion Outfit Composition (FOC) requires sequentially assembling fashion items into a stylistically cohesive ensemble. Existing works struggle to model this step-by-step process effectively, primarily because they fail to jointly optimize the two critical capabilities required for FOC: intermediate outfit value evaluation and complementary item prediction. This structural disconnect, compounded by the severe sparsity of step-wise aesthetic signals, leaves them without a mechanism to autonomously determine when to stop the composition process. To address these challenges, we introduce the Unified Aesthetic Foresight Model (UAFM), which seamlessly unifies both capabilities into a dual-head architecture over a shared backbone. Crucially, we formulate FOC as a deterministic Markov Decision Process and optimize a value head via a post-decision state temporal difference (TD) objective to recursively backpropagate sparse terminal aesthetic rewards to intermediate states. This enables the model to accurately estimate the potential of a partial outfit evolving into a compatible ensemble. Consequently, UAFM can compute step-wise marginal aesthetic gains for dynamic termination. Furthermore, this unified design strictly aligns aesthetic evaluation and complementary item prediction within a shared representation manifold, intrinsically regularizing the combinatorial search space. Extensive experiments on the Polyvore-Outfits dataset demonstrate that UAFM establishes a new state-of-the-art across both FOC and conventional fashion tasks. Extensive ablation studies further confirm that our post-decision state TD formulation provides the aesthetic value prediction necessary for autonomous dynamic termination, while validating the synergistic benefits of our unified dual-head architecture.

Figures & tables

Figure 1: Fashion Outfit Composition: Challenge and Paradigm. Conventional paradigms face severe reward sparsity during sequential composition, as positive supervision is restricted to complete human-curated outfits, leaving intermediate partial states unmonitored (Panel 1). Addressing FOC requires dual capabilities: bootstrapping intermediate state values from sparse terminal rewards and predicting next-item intent. Unifying intermediate aesthetic evaluation (Value Head) and candidate selection (Policy Head) onto a shared backbone enables autonomous dynamic termination by tracking diminishing marginal aesthetic gains (Panel 2). Our method generates cohesive ensembles, whereas conventional baselines suffer from severe pattern conflicts and item overlaps (Panel 3).

Figure 2: Overview of the UAFM architecture and training pipeline. (A) MDP Trajectory: Human-curated outfits are decomposed into state-action trajectories, governed by step penalties and terminal aesthetic rewards. (B) UAFM Architecture: A shared encoder backbone jointly parameterizes the value head (aesthetic potential) and the policy head (stylistic intent) within a dual-head framework. (C) Joint Training Objective: (C1) Value Head Training employs a proposed post-decision state TD objective to recursively backpropagate sparse terminal rewards to early intermediate states; (C2) Policy Head Training aligns latent intents via InfoNCE-based contrastive learning.

Method	H. Pref.	VAS	Cate. Align	Cate. Dup. $\downarrow$	Vis. Red. $\downarrow$	Latency (s) $\downarrow$
Random	-	2.50 ± 0.04	42.34 ± 0.15	17.84 ± 0.13	0.18 ± 0.03	0.0078
Type-Aware	-	2.69 ± 0.04	41.15 ± 0.28	21.46 ± 0.11	1.39 ± 0.08	0.15
OutfitTrans.	25.75%	3.18 ± 0.03	62.21 ± 0.23	20.58 ± 0.25	6.10 ± 0.11	0.14
Gemini-3.5	28.08%	3.28 ± 0.03	64.93 ± 0.23	15.48 ± 0.17	8.60 ± 0.15	6.35
UAFM	46.17%	3.57 ± 0.02	72.18 ± 0.27	5.69 ± 0.15	1.20 ± 0.09	0.13

Table 1: Results of composed outfits quality across different baselines. Bold and underline indicate the best and second-best performances, respectively. Metrics with

\downarrow

denote lower values are better. Scores for VAS , Cate. Align (%) , Cate. Dup. (%) , and Vis. Red. (%) report 5-fold Mean

\pm

SE.

Figure 3: Radar chart of VLLM Aesthetic Scores (VAS). Composed outfits are evaluated across five aesthetic dimensions along with the overall average score.

Method	AUC	FITB (%)	R@10 (%)	R@30 (%)	R@50 (%)
Type-Aware	0.84	55.65	3.66	8.26	11.98
SCE-Net	-	53.67	4.41	9.85	13.87
CSA-Net	0.87	59.26	5.93	12.31	17.85
LLM-DPO	0.81	61.00	-	-	-
OutfitTransformer	0.90	69.01	10.51	20.33	27.64
UAFM (Ours)	0.93	70.84	12.05	22.63	29.50

Table 2: Performance comparison on downstream discriminative tasks on disjoint version of the Polyvore-Outfits dataset. Best results are highlighted in bold .

Figure 4: Ablation on training objectives. Part (a) compares average composed outfit length (blue bars) against GT outfit length (red dashed line) and natural stop rate (green bars). Part (b) evaluate predicted aesthetic scores across composition steps with 3-fold cross-validation error bars.

Architecture	Composed Outfit Quality				Discriminative Performance
Architecture	VAS	Cate. Align %	Cate. Dup. % $\downarrow$	Vis. Red. % $\downarrow$	AUC	FITB	R@10	R@30	R@50
BCE	3.33 ± 0.02	55.23 ± 0.27	15.89 ± 0.11	3.70 ± 0.07	0.96	70.41	11.77	20.10	29.03
Separate	3.27 ± 0.02	67.73 ± 0.29	18.01 ± 0.32	6.75 ± 0.22	0.84	68.57	10.09	19.72	26.25
UAFM	3.60 ± 0.02	78.20 ± 0.49	1.29 ± 0.07	0.34 ± 0.04	0.93	70.84	12.05	22.63	29.50

Table 3: Performance comparison among the BCE variant, the Separate variant, and UAFM . All models share the identical backbone and training recipe, operating under dynamic termination without explicit length constraints. Best results are highlighted in bold .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Strategy	Latency	Cate. Align % $\uparrow$	Cate. Dup. % $\downarrow$	Vis. Red. % $\downarrow$	VLLM Aesthetic Score $\uparrow$
Strategy	Latency	Cate. Align % $\uparrow$	Cate. Dup. % $\downarrow$	Vis. Red. % $\downarrow$	Col.	Bal.	Sty.	Tex.	Wear.	Overall
Value-Greedy	0.028s	78.99 ± 0.48	1.01 ± 0.07	0.37 ± 0.02	3.93	3.47	3.77	3.24	3.52	3.59 ± 0.02
Reranking	0.24s	78.80 ± 0.43	1.07 ± 0.06	0.39 ± 0.06	3.93	3.48	3.78	3.31	3.48	3.60 ± 0.03
Beam Search	0.13s	78.20 ± 0.49	1.29 ± 0.07	0.34 ± 0.04	3.94	3.50	3.77	3.30	3.50	3.60 ± 0.03
Policy-Only	0.021 s	53.86 ± 0.14	6.82 ± 0.21	2.78 ± 0.13	3.91	3.26	3.68	3.26	3.03	3.43 ± 0.03

Appendix

Table 4: Comparison of different decoding strategies utilizing UAFM for outfit composition.

Figure 5: Interface screenshot of the Human Preference A/B testing questionnaire on Microsoft Forms. Evaluators conduct blind pairwise comparisons between outfits composed by different algorithms under standardized visual criteria.

Figure 6: Qualitative comparison of composed outfits across different paradigms. Given an initial seed garment, each system sequentially retrieves complementary items to assemble a complete ensemble. Compared with baseline paradigms, UAFM (Ours) achieves superior stylistic cohesion, color harmony, and category diversity, closely resembling human compositions ( Ground Truth ).

FashionStylist: An Expert Knowledge-enhanced Multimodal Dataset for Fashion Understanding
Apr 10, 2026Kaidong Feng, Zhuoxuan Huang, Huizhong Guo +7Multimodal Understanding
Yanshan University Qinhuangdao, China · Central South University Changsha, China · Zhejiang University Hangzhou, China +3
OmniFashion: Towards Generalist Fashion Intelligence via Multi-Task Vision-Language Learning
Mar 3, 2026Zhengwei Yang, Andi Long, Hao Li +3Vision-Language Models Large Vision-Language Models
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University · Harbin Institute of Technology
VersaVogue: Visual Expert Orchestration and Preference Alignment for Unified Fashion Synthesis
Apr 8, 2026Jian Yu, Fei Shen, Cong Wang +4Image Generation Controllable Image Generation
Nanjing University of Science and Technology · National University of Singapore · Nanjing University +1

Apr 10, 2026J/K move · Enter open · S save

FashionStylist: An Expert Knowledge-enhanced Multimodal Dataset for Fashion Understanding

Kaidong Feng, Zhuoxuan Huang, Huizhong Guo, Yuting Jin+6

Yanshan University Qinhuangdao, China · Central South University Changsha, China · Zhejiang University Hangzhou, China+3

Multimodal Understanding

Open paper AskPDF arXiv

Fashion understanding requires both visual perception and expert-level reasoning about style, occasion, compatibility, and outfit rationale. However, existing fashion datasets remain fragmented and task-specific, often focusing on item attributes, outfit co-occurrence, or weak textual supervision, and thus provide limited support for holistic outfit understanding. In this paper, we introduce FashionStylist, an expert-annotated benchmark for holistic and expert-level fashion understanding. Constructed through a dedicated fashion-expert annotation pipeline, FashionStylist provides professionally grounded annotations at both the item and outfit levels. It supports three representative tasks: outfit-to-item grounding, outfit completion, and outfit evaluation. These tasks cover realistic item recovery from complex outfits with layering and accessories, compatibility-aware composition beyond co-occurrence matching, and expert-level assessment of style, season, occasion, and overall coherence. Experimental results show that FashionStylist serves not only as a unified benchmark for multiple fashion tasks, but also as an effective training resource for improving grounding, completion, and outfit-level semantic evaluation in MLLM-based fashion systems.

Abstract

Explore similar work

FashionStylist: An Expert Knowledge-enhanced Multimodal Dataset for Fashion Understanding

OmniFashion: Towards Generalist Fashion Intelligence via Multi-Task Vision-Language Learning

VersaVogue: Visual Expert Orchestration and Preference Alignment for Unified Fashion Synthesis