ClothCode: Generative Modeling for Garment State Estimation and Dynamics Prediction

Yingdong Ru; Lipeng Zhuang; Shiyu Fan; Zhuo He; Florent P. Audonnet; Gerardo Aragon Camarasa

cs.ROOct 9, 2026

ClothCode: Generative Modeling for Garment State Estimation and Dynamics Prediction

Authors: Yingdong Ru, Lipeng Zhuang, Shiyu Fan, Zhuo He, Florent P. Audonnet, Gerardo Aragon Camarasa

Organizations: The authors are with the University of Glasgow, Glasgow, United Kingdom.

Abstract

We introduce ClothCode, a unified generative framework for garment state estimation and action-conditioned dynamics prediction in a shared discrete configuration space. A material-aligned tokenizer encodes garment configuration as residual vector-quantized patch tokens, while task-specific Masked Generative Transformers predict these tokens through parallel confidence-guided refinement. Across three garment templates, ClothCode reduces mean displacement error by 26.6% for state estimation and 37.0% for dynamics prediction relative to the strongest baselines. Dynamics prediction requires only 16.7 ms per window, compared with 556.2 ms for diffusion. Integrated into the same model-predictive controller, ClothCode improves real-world folding and flattening success from 56.7% to 78.9%. Further results and videos are available at: https://anonymous.4open.science/r/ClothCode-DE32

Figures & tables

Fig. 1: Overview of our discrete generative framework for model-based garment manipulation. A garment configuration is encoded as a set of discrete patch tokens shared by two Masked Generative Transformers (MGTs). (1) The discrete garment representation captures local deformation states over fixed material patches. (2) For state estimation, an MGT iteratively predicts the complete token configuration from a partial point cloud. (3) For dynamics prediction, an action-conditioned MGT predicts future garment configurations for candidate actions, allowing MPC to select the trajectory with the lowest goal cost. (4) The resulting perception–prediction loop enables closed-loop garment manipulation on a real robot.

Fig. 2: Overview of ClothCode. ClothCode first learns a discrete representation of garment shape (top). Each mesh is divided into fixed patches that track the same material regions across deformations, encoded as two-level residual vector-quantization (RVQ) tokens, and reconstructed by a shared mesh decoder. This frozen representation provides a common prediction space for two tasks. For state estimation (bottom left), a masked transformer combines a partial point cloud with masked garment tokens to recover the complete garment configuration and its global position. For dynamics prediction (bottom right), a second masked transformer combines recent garment states with a candidate end-effector trajectory and jointly predicts the future garment sequence and its global motion. $\mathrm{M}$ denotes a masked token.

	ID		Action-OOD		Physics-OOD		Rollout
Method	MDE $\downarrow$	EMD $\downarrow$	MDE $\downarrow$	EMD $\downarrow$	MDE $\downarrow$	EMD $\downarrow$	Oracle $\downarrow$	Free $\downarrow$
Fabric
UCD-Transformer [ 5 ]	1.652	1.511	2.138	1.965	1.987	1.741	1.306	2.684
UniClothDiff	1.172	1.118	1.508	1.465	1.379	1.328	1.920	3.916
PGND	1.345	1.150	1.702	1.495	1.545	1.344	1.499	2.957
ClothCode (Ours)	0.730	0.694	0.884	0.843	0.859	0.818	0.864	1.822

TABLE I: Action-conditioned dynamics prediction (cm,

\downarrow

). Single-window results predict 10 future frames from 3 context frames. Rollout MDE is reported on ID with oracle and predicted ( free ) context.

Fig. 3: Qualitative comparison of garment dynamic prediction. Per-vertex reconstruction errors with respect to the ground-truth mesh are visualized using a shared color scale across all methods.

Fig. 4: Qualitative comparison of garment state estimation from camera observations. Per-vertex reconstruction errors with respect to the ground-truth mesh are visualized using a shared color scale across all methods.

		Eval A (ID)			Eval B (Action-OOD)
Garment	Method	MDE $\downarrow$	EMD $\downarrow$	Chamfer $\downarrow$	MDE $\downarrow$	EMD $\downarrow$	Chamfer $\downarrow$
Fabric	Uniclothdiff	1.384	1.205	1.770	1.624	1.352	1.896
	Transformer	1.910	1.388	1.881	2.285	1.565	2.021
	GarmentNets	1.993	1.745	2.314	2.191	1.870	2.423
	MEDOR	2.106	1.809	2.343	2.290	1.918	2.417
	ClothCode (Ours)	0.890	0.680	1.182	1.104	0.795	1.279

TABLE II: Garment state estimation from partial point-cloud observations.

State + Dynamics	Fabric F/U	T-shirt F/U	Sweater F/U
ClothCode + Uniclothdiff	83.3/76.7	66.7/70.0	50.0/56.7
Uniclothdiff + ClothCode	73.3/73.3	70.0/70.0	60.0/66.7
Uniclothdiff + Uniclothdiff	66.7/66.7	56.7/60.0	40.0/50.0
ClothCode + ClothCode (Ours)	90.0/80.0	80.0/80.0	66.7/76.7

TABLE III: Real-world MPC success rates (%). Each garment column reports folding/flattening (F/F), respectively.

Fig. 5: Qualitative results of real-world experiments.

Representation	Prediction	Dynamics $\downarrow$	State $\downarrow$
Continuous global	One-shot regression	$1.685\pm 0.028$	$2.093\pm 0.071$
Continuous patch latent	One-shot regression	$1.327\pm 0.015$	$2.673\pm 0.070$
Continuous patch latent	Iterative diffusion	$1.014\pm 0.032$	$2.105\pm 0.170$
Discrete RVQ	Trained one-shot	$1.300\pm 0.019$	$1.996\pm 0.073$
Discrete RVQ	Masked iterative	$\mathbf{0.674\pm 0.023}$	$\mathbf{1.211\pm 0.075}$

TABLE IV: Ablation of representation and prediction design. MDE is reported in cm; lower is better.

Generative Modeling

Generative modeling learns data distributions to create new samples, from images and text to molecules, simulations, and structured records.

See all 487 papers

MeshPriorDiT: Hierarchical Modeling for Action-Conditioned Cloth Dynamics
Aug 27, 2026Zihang Wang, Jianming Hu, Shang Su +4Deformable Object Manipulation Diffusion Models for Motion Generation
Tsinghua University · Dense-AI · University of Michigan
DiT-Garment: Garment Dynamics with Diffusion Transformers
Sep 16, 2026Antoine Dumoulin, Laurence Boissieux, Joao Regateiro +23D Mesh Deformation Deformable Object Simulation
Inria Centre at the University Grenoble Alpes · InterDigital Inc. · Inria, University of Rennes, CNRS, IRISA-UMR 6074
ClothTransformer: Unified Latent-Space Transformers for Scalable Cloth Simulation
May 27, 2026Yu Zhang, Yidi Shao, Wenqi Ouyang +5Transformer Physics-Based Simulation
1S-Lab, Nanyang Technological University, Singapore · 2Feeling AI · University of Oxford +2

Aug 27, 2026J/K move · Enter open · S save

MeshPriorDiT: Hierarchical Modeling for Action-Conditioned Cloth Dynamics

Zihang Wang, Jianming Hu, Shang Su, Hao Huang+3

Tsinghua University · Dense-AI · University of Michigan

Deformable Object Manipulation Diffusion Models for Motion Generation

Open paper AskPDF arXiv

Action-conditioned cloth dynamics prediction requires both locally plausible deformation and long-range coordination. Existing approaches largely follow two paradigms. Mesh-based GNNs capture local physical responses through material connectivity. However, their finite message-passing range limits coordination between topologically distant regions, while autoregressive rollouts tend to accumulate prediction errors. Transformer-based dynamics models capture long-range interactions through global attention, but often operate without explicit material connectivity and must learn local topological responses directly from data. We propose MeshPriorDiT, a hierarchical dynamics model that decomposes future cloth motion into a structured mesh prior and a generative residual. An action-conditioned mesh GNN first predicts multi-step vertex displacements, yielding a reference trajectory that respects material topology and grasp constraints. Conditioned on historical states, planned actions, and the mesh prior, a Residual DiT then uses conditional flow matching to jointly generate the residual motion not captured by the prior. The generated residual is further rescaled and decoded using material adjacency to coordinate corrections across neighboring vertices. We evaluate MeshPriorDiT on 15-step autoregressive rollouts across three cloth manipulation tasks. Averaged over the three tasks, MeshPriorDiT reduces average Global MSE by 43.42% relative to the GNN-Only baseline and by 75.03% relative to the DiT-DDPM baseline, while maintaining a favorable Edge-strain MSE comparable to that of GNN-Only.

Abstract

Generative Modeling

Explore similar work

MeshPriorDiT: Hierarchical Modeling for Action-Conditioned Cloth Dynamics

DiT-Garment: Garment Dynamics with Diffusion Transformers

ClothTransformer: Unified Latent-Space Transformers for Scalable Cloth Simulation