Multi-View Transformer with Geometry-Guided Multi-Modal ControlNet for 3D Virtual Try-On
Organizations: University of Surrey, Guildford, UK
Abstract
Editing a 3D Gaussian Splatting (3DGS) capture of a person so that they wear a new garment requires every rendered view to agree on the same garment details. Existing 3D virtual try-on (VTON) pipelines, however, edit training views largely independently and fuse them with hand-designed heuristics such as manually chosen reference views and fixed attention-blending weights. The resulting inconsistencies are baked into the 3D representation and surface as texture drift, blurred prints, and implausible geometry in novel views. We present GS-VTON-MVT, a three-stage framework that combines implicit and explicit 3D reasoning for consistent 3DGS try-on. (i) A multi-view inpainting transformer jointly edits a set of training views with full cross-view self-attention, so that cross-view correspondence is learned rather than imposed. (ii) A geometry-guided multi-modal ControlNet conditions the diffusion editor on the garment image together with depth and surface normals rendered from the current 3DGS state, supplying explicit structure where visual evidence is weakest. (iii) A pose-adaptive persona-aware attention replaces the fixed blending weight between current-view and reference-view features with weights derived from camera-pose proximity. On the 3D-VTONBench benchmark, GS-VTON-MVT improves FID, KID, LPIPS-based cross-view consistency, and user preference over general-purpose 3D editors and prior 3D try-on methods.
Figures & tables
| Method | FID | LPIPS | KID |
|---|---|---|---|
| Instruct-NeRF2NeRF [ 15 ] | 52.3 | 0.231 | 0.041 |
| Instruct-GS2GS [ 37 ] | 47.8 | 0.198 | 0.035 |
| GaussianEditor [ 8 ] | 44.2 | 0.176 | 0.029 |
| GaussCtrl [ 43 ] | 41.5 | 0.165 | 0.026 |
| ViCA-NeRF [ 12 ] | 43.1 | 0.171 | 0.028 |
| GS-VTON [ 5 ] | 38.2 | 0.142 | 0.023 |
| Variant | FID | LPIPS | KID |
|---|---|---|---|
| A: Baseline (GS-VTON) [ 5 ] | 38.2 | 0.142 | 0.023 |
| B: A + MVT | 32.1 | 0.118 | 0.017 |
| C: A + Multi-modal ControlNet | 34.5 | 0.125 | 0.019 |
| D: A + Pose-adaptive attention | 36.8 | 0.135 | 0.021 |
| E: A + MVT + Multi-modal ControlNet | 27.3 | 0.092 | 0.011 |
| F: Full model | 24.1 | 0.076 | 0.008 |
| Method | Realism | Garment sim. | Overall |
|---|---|---|---|
| Instruct-NeRF2NeRF [ 15 ] | 8 | 6 | 7 |
| Instruct-GS2GS [ 37 ] | 12 | 10 | 11 |
| GaussianEditor [ 8 ] | 10 | 12 | 11 |
| GaussCtrl [ 43 ] | 14 | 16 | 15 |
| ViCA-NeRF [ 12 ] | 9 | 8 | 8 |
| GS-VTON [ 5 ] | 18 | 20 | 19 |
Virtual Try-On
Virtual try-on generates images or videos showing people wearing selected garments while preserving appearance, fit, pose, and identity.
See all 58 papers