Spatio-Temporal Garment Reconstruction Using Diffusion Mapping via Pattern Coordinates

Yingxuan You; Ren Li; Corentin Dumery; Cong Cao; Hao Li; Pascal Fua

Fig. 2: Pipeline . Given an image of a clothed person, we first estimate the front normal

\mathbf{n}_{F}

of the target garment, and the SMPL body model which is used to render the body part segmentation (

\mathbf{s}_{F}

,

\mathbf{s}_{B}

) and depth (

\mathbf{d}_{F}^{b}

,

\mathbf{d}_{B}^{b}

) images. The back normal

\mathbf{n}_{B}

of the garment is estimated subsequently by the diffusion model

\boldsymbol{\epsilon}_{\theta}^{n}

. We then predict the UV-coordinate (

\mathbf{c}_{F}

,

\mathbf{c}_{B}

) and the depth (

\mathbf{d}_{F}^{g}

,

\mathbf{d}_{B}^{g}

) images from the garment normal and body estimations with the mapping model

\boldsymbol{\epsilon}_{\theta}^{m}

. The incomplete UV positional map

\tilde{\mathcal{U}}

is produced from them using the camera backprojection. Finally, we fit

\tilde{\mathcal{U}}

to DISP to recover the complete UV positional map

\hat{\mathcal{U}}

and the corresponding garment mesh

\mathbf{g}

, which is further improved by the refinement.

Fig. 3: Mapping between pixel, 3D, and UV spaces. The pixel

(x,y)

is mapped to

(X,Y,Z)

in the 3D space using the estimated depth

d

and the camera backprojection

P^{-1}

, and to

(u,v)

in the UV space using the estimated UV coordinates

(u,v,\sigma)

. The dash line indicates that

(X,Y,Z)

and

(u,v)

are connected indirectly through

(x,y)

.

Fig. 4: Recovering garment rest geometry . Given (a) the incomplete panel mask

\tilde{\mathcal{M}}

, we fit (b) the complete panel mask

\mathcal{M}

by Eq. 11 . (c) shows the overlay of

\tilde{\mathcal{M}}

in gray and

\mathcal{M}

in white. (d) is the corresponding rest-state garment mesh

\bar{\mathbf{g}}

for (b).

Fig. 5: Processing a video sequence. Given a set of images with extracted body segmentations

\mathbf{S}

, body depths

\mathbf{D}^{b}

, and garment normals

\mathbf{N}^{F}

, our method produces a sequence of garment meshes

\mathbf{G}

in three steps. First, the back-view normals

\mathbf{N}^{B}

of the garment are inferred. By design, our method ensures these predictions are temporally consistent. Second, a mapping network estimates the 2D/3D positions of each pixel, where the 2D positions

(\mathbf{C}_{F},\mathbf{C}_{B})

are in a reference pattern space, and the 3D positions are represented as depth maps

(\mathbf{D}_{F}^{g},\mathbf{D}_{B}^{g})

. We introduce novel guidance on this generation to match normal estimations and prevent intersection with the body. Finally, this mapping is unwrapped into a partial 2D pattern

\tilde{\mathbf{U}}

where pixel value encodes the 3D position, and our temporal inpainting diffusion completes these partial observations into a full garment sequence while ensuring the partial constraints are respected.

Fig. 6: Architecture of the spatio-temporal diffusion model. The model decouples spatial and temporal knowledge into separate modules: the spatial module (red) captures per-frame spatial structure, while the temporal module (blue) models per-pixel motion across frames. Together, these components form the foundation for high-fidelity and temporally smooth 4D garment reconstruction.

Method	Skirt			Trousers			Tshirt			Jacket
Method	CD $\downarrow$	NC $\uparrow$	IoU $\uparrow$	CD $\downarrow$	NC $\uparrow$	IoU $\uparrow$	CD $\downarrow$	NC $\uparrow$	IoU $\uparrow$	CD $\downarrow$	NC $\uparrow$	IoU $\uparrow$
SMPLicit [ 11 ]	5.80	0.34	53.72	1.46	-0.12	74.53	2.77	0.09	58.64	2.40	0.03	54.81
ISP [ 45 ]	4.18	0.78	68.10	1.53	0.78	75.39	1.36	0.82	68.70	1.69	0.79	71.31
GaRec $\dagger$ [ 44 ]	2.67	0.88	96.33	1.67	0.85	92.55	1.19	0.83	87.15	1.21	0.79	91.40
D 3 -Human $\dagger$ [ 8 ]	3.64	0.78	76.42	1.80	0.79	82.49	1.79	0.79	78.51	1.71	0.76	56.60
DMap-Static (Ours)	2.26	0.89	88.80	1.36	0.84	86.23	1.06	0.83	80.20	1.29	0.83	78.28

TABLE I: Quantitative comparison with SOTA methods. “

\dagger

” denotes models that use refinement. Bold : best; Underline : second best.

Fig. 7: Qualitative comparison on an in-the-wild image. The top and bottom rows show the front and the back of the reconstructions produced by our method DMap, BCNet [ 30 ] , SMPLicit [ 11 ] , ISP [ 45 ] , GaRec [ 44 ] and ECON [ 72 ] , respectively.

Fig. 8: Qualitative comparison on a video sequence. Our method (second row) achieves finer geometric detail, better temporal consistency, and avoids the garment–body collisions frequently observed in REC-MV [ 59 ] (bottom row), where the reconstructed garment often penetrates the body surface.

Fig. 9: Reconstruction for in-the-wild images. Our method can recover realistic 3D models for diverse garments in different shapes and deformations.

Fig. 10: Qualitative results on real-world videos. For each sample, we show the input, the front- and back-view reconstructions, and the 2D renderings.

Fig. 11: Ablation of fitting strategy . (a) The input image and its normal estimations for the front and back. (b) Our full reconstruction. (c) Reconstruction without post-refinement. (d) Reconstruction refined by optimizing only the neural displacement field. (e) Reconstruction refined by optimizing only the vertex positions. (f) Reconstruction using only the front normal.

	Method	Time/Frame	Hardware	VRAM Usage	I/O
Image-based	SMPLicit [9]	8 min	A100-80G	3.8 GB	1/1
	ISP [14]	19 min	A100-80G	2.3 GB	1/1
	GaRec $\dagger$ [15]	1 min	A100-80G	12.0 GB	1/1
	DMap-Static	7 min	A100-80G	6.8 GB	1/1
	DMap-Static $\dagger$	24 min	A100-80G	6.8 GB	1/1
Video-based	REC-MV $\dagger$ [16]	12 min	A100-80G	19.6 GB	1/1

TABLE II: Inference time. “

\dagger

” denotes models that use refinement.

Fig. 12: Acceleration comparison. “Inf.”: inference; “Ref.”: refinement.

Method	Mean	Median	Min	Max
DMap-Static (Inf.)	18.232	11.994	4.546	85.132
DMap-Static (Inf. + Ref.)	25.470	21.355	10.219	93.281
DMap-Dynamic (Inf.)	5.588	3.939	0.945	23.843
DMap-Dynamic (Inf. + Ref.)	3.497	2.122	0.869	13.967
GT	3.724	2.846	0.622	13.193

TABLE III: Quantitative acceleration statistics.

Method	Skirt			Trousers			Tshirt			Jacket
Method	CD $\downarrow$	NC $\uparrow$	IoU $\uparrow$	CD $\downarrow$	NC $\uparrow$	IoU $\uparrow$	CD $\downarrow$	NC $\uparrow$	IoU $\uparrow$	CD $\downarrow$	NC $\uparrow$	IoU $\uparrow$
inference w/o back view	2.67	0.90	83.62	1.90	0.81	79.09	1.61	0.81	78.31	2.00	0.73	70.46
inference w/o $\mathcal{G}_{\mathrm{D2N}}$	1.97	0.92	90.67	1.53	0.82	86.53	1.24	0.84	85.73	1.56	0.82	72.55
inference w/o $\mathcal{G}_{\mathrm{inter}}$	1.71	0.92	90.53	1.30	0.82	86.62	1.07	0.83	88.23	1.24	0.83	77.63
inference	1.62	0.93	90.93	1.21	0.83	88.72	0.92	0.86	91.95	1.11	0.85	81.75
inference + refinement	1.54	0.92	96.73	0.83	0.87	95.41	0.76	0.92	96.46	0.93	0.86	93.47

TABLE IV: Ablation study of the proposed spatial guidance in DMap-dynamic on the CLOTH3D dataset.

Fig. 13: Ablation on spatial guidance. (a) 2D observations, including input image, front normal, and back normal. (b) Reconstruction after refinement. (c) Reconstruction after inference. (d) Inferred result without using generated back-view normals. (e) Inferred result without depth-to-normal guidance

\mathcal{G}_{D2N}

. (f) Inferred result without interpenetration-aware guidance

\mathcal{G}_{inter}

.

Figure 17

Figure 18

Fig. 18: Qualitative results under profile views, occlusions, cropping, and out-of-distribution garment types.

Fig. 19: Qualitative results on a profile-view and occluded video sequence.

Fig. 20: Reconstruction results on multi-layered structures.

Fig. A: Qualitative comparison with the pattern-draping-fitting method on images.

Fig. B: Qualitative comparison with the pattern-draping-fitting method on a video sequence.

Fig. C: Comparison between DMap-Static and GaRec [ 43 ] .

Fig. D: More reconstruction results for in-the-wild images. DMap-Static can handle both the tight-fitting and the loose-fitting garments and recover high-fidelity 3D meshes for them.

Fig. E: Additional qualitative comparison for in-the-wild videos. DMap-Dynamic achieves both geometric accuracy and temporal consistency.

Fig. F: Visualization of error distribution on the reconstructed jacket mesh. The bright colors indicate large errors.

Fig. G: Visualization of error distribution on the reconstructed skirt mesh. The bright colors indicate large errors.

Fig. H: Recovering the rest garment using accumulated partial masks in DMap-Dynamic .

Eq. 11	$\lambda_{area}=0.5,\lambda_{\mathbf{z}}=0.02$
Eq. 12	$\rho=20$
Eq. 14	$\lambda_{m}=1,\lambda_{d}=2,\lambda_{n}=2,\lambda_{u}=100,\lambda_{p}=1$

TABLE A: The values of hyper-parameters.

	CD $\downarrow$	IoU $\uparrow$	NC $\uparrow$
w/o $N_{B}$	1.67	90.04	0.78
w/o $f_{\phi}$ , w/o $V$	1.45	92.67	0.81
w/o $f_{\phi}$ , w/ $V$	1.32	93.34	0.82
w/ $f_{\phi}$ , w/o $V$	1.27	94.71	0.82
Ours	1.21	95.32	0.83

TABLE B: Ablation study. w/o

N_{B}

means optimization without using the back normal estimation

N_{B}

.

f_{\phi}

denotes optimization using the neural displacement field, and

V

denotes optimization conducted directly on vertex positions.

Fig. I: Clothing looseness variation. Columns correspond to garment categories from left to right: T-shirt, trousers, jacket, and skirt. In each column, examples of the same garment category are ordered from top to bottom from tight-fitting to loose-fitting.

Fig. J: Loose-garment reconstruction from Internet videos . First row: loose-fitting T-shirt and trousers. Second row: skirt.

Fig. K: CLOTH3D dataset on the 33 garments in each category.

Fig. L: Re-simulated CLOTH3D garments using AMASS motion sequences.

Spatio-Temporal Garment Reconstruction Using Diffusion Mapping via Pattern Coordinates

Abstract

Figures & tables

3D Garment Reconstruction

Stitched Embeddings: A Unified Latent Space for 3D Garments and 2D Patterns

OmniFabric: Coherent UV Space Texture Synthesis for 3D Garment Reconstruction

Forwardrobe: Garment-Aware Gaussian Avatars from a Single Image

Abstract

3D Garment Reconstruction

Explore similar work

Stitched Embeddings: A Unified Latent Space for 3D Garments and 2D Patterns

OmniFabric: Coherent UV Space Texture Synthesis for 3D Garment Reconstruction

Forwardrobe: Garment-Aware Gaussian Avatars from a Single Image