InstructVVT: Instruction-Driven Video Virtual Try-On without Auxiliary Spatial Priors

Dingbao Shao; Song Wu; Xinyu Chen; Qian Wang; Jiahang Li; Kuai Jiang; Jiang Lin; Yuhang Liu

cs.CVAug 14, 2026

InstructVVT: Instruction-Driven Video Virtual Try-On without Auxiliary Spatial Priors

Authors: Dingbao Shao, Song Wu, Xinyu Chen, Qian Wang, Jiahang Li, Kuai Jiang, Jiang Lin, Yuhang Liu,

Abstract

Video virtual try-on is a highly constrained editing task requiring the precise replacement of a target person's clothing while strictly preserving the original video's spatial structure and temporal dynamics. Existing methods heavily rely on auxiliary handcrafted spatial priors (e.g., masks, poses) for editing control. However, these priors are prone to failure in unconstrained real-world videos and often compress rich visual context into incomplete structural signals. Furthermore, standard reconstruction objectives fail to fully capture try-on-specific human preferences. To address these challenges, we propose InstructVVT, an instruction-driven and reference-guided video virtual try-on framework based on a Diffusion Transformer (DiT) that operates without inference-time spatial priors. Our core insight is to recover fine-grained control directly from the input triplet (source video, reference garment, and instruction) via a dual-level reference conditioning scheme. Specifically, an MLLM infers semantic edit tokens for target disambiguation and structural preservation, while a lightweight conditioning pathway explicitly injects fine-grained visual garment details. Finally, we design a try-on-specific reward and utilize the DiffusionNFT algorithm to align the model with human preferences. Extensive experiments on ViViD-S and TripVVT-Bench demonstrate that InstructVVT outperforms state-of-the-art open-source methods in garment fidelity, structural preservation, and temporal consistency, despite requiring fewer inference-time controls.

Figures & tables

Figure 1: Teaser examples of instruction-driven video virtual try-on. Given a source video, a reference garment, and a natural-language instruction, the model edits the instructed target person while preserving the source identity, motion, background, and temporal consistency.

Figure 2: Overview of our InstructVVT framework. In the supervised training, the source video, reference garment, and instruction are encoded by a frozen MLLM with trainable query tokens $Q_{e}$ , LoRA adapters ( Hu et al., 2022 ) , and an MLP connector to produce MLLM edit tokens $C_{e}$ . We adopt two embedding layers with identical architecture but independent parameters to encode video cue tokens $X_{t}$ and reference garment tokens $C_{g}$ , respectively. The MLLM edit tokens $C_{e}$ , video cue tokens $X_{t}$ and reference garment tokens $C_{g}$ are injected into each DiT block through cross-attention. The post-training stage applies DiffusionNFT ( Zheng et al., 2025 ) : a frozen EMA policy samples $K$ videos, a tailored reward model based on MLLM assigns score-token rewards, and the NFT loss updates the trainable DiT.

Method	VFID I $\downarrow$	VFID R $\downarrow$	CLIP-F $\uparrow$	BG-L1 $\downarrow$	BG-DINO $\downarrow$	MLLM-Reward $\uparrow$
ViViD ( Fang et al., 2024 )	21.8032	0.8212	0.9574	0.0571	0.0055	0.6341
CatV2TON ( Chong et al., 2025 )	19.5131	0.5283	0.9304	0.0507	0.0071	0.5773
MagicTryOn ( Li et al., 2025a )	17.5710	0.5073	0.9647	0.0493	0.0056	0.5199
TripVVT ( Shao et al., 2026 )	16.4398	0.3315	0.9670	0.0624	0.0059	0.6034
UniVideo ( Wei et al., 2025 )	23.6970	1.2139	0.9965	0.1447	0.0124	0.7152
Ours	16.0271	0.2809	0.9957	0.0267	0.0048	0.7236

Table 1: Quantitative comparison with other methods on ViViD-S ( Chong et al., 2025 ) test dataset. We additionally report the proposed MLLM-Reward score. Best results are shown in bold , and second-best results are underlined .

Method	VFID ${}_{I}\downarrow$	VFID ${}_{R}\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CLIP-I $\uparrow$	CLIP-F $\uparrow$	BG-L1 $\downarrow$	BG-DINO $\downarrow$	MLLM-Reward $\uparrow$	User Pref. $\uparrow$
ViViD ( Fang et al., 2024 )	26.7620	0.7083	0.8323	0.1343	0.8716	0.9849	0.0691	0.0053	0.5569	–
CatV2TON ( Chong et al., 2025 )	34.2275	3.3266	0.7845	0.2130	0.8077	0.9819	0.1032	0.0109	0.2811	–
MagicTryOn ( Li et al., 2025a )	22.9238	0.5502	0.8641	0.1150	0.9102	0.9881	0.0515	0.0036	0.5979	5.4%
TripVVT ( Shao et al., 2026 )	20.7245	0.3163	0.8538	0.1053	0.9373	0.9876	0.0852	0.0059	0.6330	–
UniVideo ( Wei et al., 2025 )	28.4413	3.0811	0.5461	0.1871	0.9172	0.9897	0.2659	0.0091	0.6915	1.6%
Kling 1.6 ( Kuaishou Technology, 2025 )	22.4242	0.6289	0.6662	0.1353	0.9311	0.9504	0.1662	0.0064	0.7227	13.6%

Table 2: Quantitative comparison with other methods on TripVVT-Bench. User preference reports first-place vote ratios from the four-method shortlist study; methods outside the shortlist are marked by “–”. Best results are shown in bold , and second-best results are underlined ; ties share the same rank, with the next distinct value marked as second-best when applicable.

Metric	Pairwise agreement $\uparrow$	Spearman $\rho$ $\uparrow$	Kendall $\tau$ $\uparrow$
Ours MLLM reward	65.5%	0.368	0.310

Table 3: Reward-human agreement on TripVVT-Bench. Agreement is computed on 100 test samples with four generated videos per sample.

Variant	VFID ${}_{I}\downarrow$	VFID ${}_{R}\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CLIP-I $\uparrow$	CLIP-F $\uparrow$	BG-L1 $\downarrow$	BG-DINO $\downarrow$	MLLM-Reward $\uparrow$
w/o MLLM, using T5	18.3755	0.3496	0.8467	0.0808	0.9320	0.9882	0.0584	0.0053	0.6581
w/o garment input to MLLM	13.0479	0.1161	0.8717	0.0633	0.9514	0.9881	0.0487	0.0048	0.7064
w/o source-video condition	14.5686	0.1938	0.8659	0.0673	0.9476	0.9878	0.0538	0.0049	0.7068
w/o reference-garment tokens	14.8048	0.1996	0.8681	0.0675	0.9438	0.9881	0.0520	0.0048	0.6321
w/o DiffusionNFT post-training	13.8390	0.1381	0.8688	0.0656	0.9482	0.9882	0.0490	0.0048	0.6859
Full model	12.8589	0.1404	0.8742	0.0615	0.9528	0.9881	0.0468	0.0047	0.7193

Table 4: Ablation studies on TripVVT-Bench. We use the same metric setting as the TripVVT-Bench quantitative comparison. Best results are shown in bold , and second-best results are underlined ; ties share the same rank, with the next distinct value marked as second-best when applicable.

Figure 3: Qualitative results on TripVVT-Bench. Our method preserves the source video structure and transfers the reference garment to the instructed subject with better temporal consistency.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: A limitation case under conflict between text semantics and the reference garment image. When the instruction describes garment attributes that are inconsistent with the visual reference, the model may not fully preserve the reference garment appearance.

Benchmark	Videos	Frames	Resolution	Instruction source
ViViD-S	180	61	$832{\times}624$	MLLM-generated
TripVVT-Bench	100	49	$832{\times}480$	MLLM-generated

Appendix

Table 5: Evaluation benchmarks and inference settings.

Stage	Objective	Data	Trainable	Steps	Batch	EMA	Hardware
1	Interface alignment	Mixed	Q, LoRA, Conn	20K	1	–	8 H100
2	Full-condition SFT	Mixed	MLLM, DiT	25K	1	–	8 H100
3	High-resolution SFT	TripVVT-10K	MLLM, DiT	15K	1	0.999	8 H100
RL	DiffusionNFT	Rollouts	Gen	180	30	0.5	40 H100

Appendix

Table 6: Training and compute settings.

Figure 5: Reward trajectory during DiffusionNFT RL post-training. The curve tracks the MLLM reward used to optimize the video generator in the RL stage.

Asset	Category	Role	License / terms
VITON-HD	Dataset	Image source persons	CC BY-NC 4.0
DressCode	Dataset	Reference garments	Academic non-commercial terms
ViViD	Dataset / benchmark	Video triplets; ViViD-S	Apache-2.0
TripVVT-10K / TripVVT-Bench	Dataset / benchmark	Training/evaluation	Official release terms
CatVTON	Method / baseline	Image target synthesis	CC BY-NC-SA 4.0
MagicTryOn	Method / baseline	Video target synthesis	CC BY-NC-SA 4.0

Appendix

Table 7: External assets used in this work and their public license or terms status.

Figure 7: Qualitative comparisons on ViViD-S. The examples show that InstructVVT transfers the reference garment while preserving the source video appearance and temporal structure, complementing the quantitative ViViD-S results in Table 1 .

Figure 8: Qualitative ablation comparison on TripVVT-Bench. Replacing the MLLM with T5 weakens video- and garment-aware edit conditioning; removing the reference garment from the MLLM input reduces high-level garment-to-person alignment; removing source-video conditioning weakens preservation; and removing DiffusionNFT post-training reduces the overall try-on quality. These visual trends are consistent with the quantitative ablations in Table 4 .

Figure 9: Additional ViViD-S visualizations. These samples provide a broader view of the method’s behavior across different source videos and reference garments, showing stable garment appearance and source preservation across sampled frames.

Figure 10: Additional qualitative comparisons on TripVVT-Bench. The examples cover diverse in-the-wild scenes, reference garments, and target-person configurations, complementing the main-paper qualitative results.

Virtual Try-On

Virtual try-on generates images or videos showing people wearing selected garments while preserving appearance, fit, pose, and identity.

See all 55 papers

UniVVT: A Unified End-to-End Framework for High-Fidelity Video Virtual Try-on
Aug 6, 2026Yushe Cao, Shikun Feng, Fei Shen +5Virtual Try-On Identity-Preserving Video Generation
Tsinghua University · Zhongguancun Academy · National University of Singapore +2
iTryOn: Mastering Interactive Video Virtual Try-On with Spatial-Semantic Guidance
May 20, 2026Jun Zheng, Zhengze Xu, Mengting Chen +6HOI Generation Virtual Try-On
Shenzhen Campus of Sun Yat-sen University · Taobao & Tmall Group of Alibaba
OmniTryOn: Video Try-On Anything at Once!
Jun 7, 2026Changliang Xia, Chengyou Jia, Minnan Luo +3Virtual Try-On Temporal Consistency in Video Generation
Xi’an Jiaotong University Xi’an, China

Aug 6, 2026J/K move · Enter open · S save

UniVVT: A Unified End-to-End Framework for High-Fidelity Video Virtual Try-on

Yushe Cao, Shikun Feng, Fei Shen, Haikuo Peng+4

Tsinghua University · Zhongguancun Academy · National University of Singapore+2

Virtual Try-On Identity-Preserving Video Generation Controllable Video Generation

Open paper AskPDF arXiv

Video Virtual Try-On (VVT) synthesizes a video of a person wearing a target garment while preserving identity, motion, and scene dynamics. Dominant approaches cast VVT as mask-conditioned video inpainting and rely on separate modules for human parsing, pose estimation, and garment warping. This multi-stage design complicates deployment and, more critically, allows errors in explicit geometric priors to propagate irreversibly into the generated video. We present UniVVT, a unified end-to-end framework that reframes VVT as semantically conditioned video generation, eliminating mask, pose, and warping modules at inference. At its core, a scene-task perceiver built on a Multimodal Large Language Model jointly encodes the source video, target garment, and task instruction into compact, task-aware latent tokens, implicitly capturing what to transfer and where and how to transfer it. A lightweight semantic bridge then aligns these tokens with the conditioning space of a diffusion-based video generator, enabling coherent garment transfer. To robustly couple the heterogeneous components, we devise a three-stage progressive training strategy comprising semantic alignment, joint task adaptation, and flexible-resolution refinement. Extensive experiments demonstrate that UniVVT achieves state-of-the-art performance across multiple benchmarks, validating implicit semantic guidance as a simple and effective alternative to fragile geometric preprocessing for end-to-end virtual try-on.

Abstract

Virtual Try-On

Explore similar work

UniVVT: A Unified End-to-End Framework for High-Fidelity Video Virtual Try-on

iTryOn: Mastering Interactive Video Virtual Try-On with Spatial-Semantic Guidance

OmniTryOn: Video Try-On Anything at Once!