Action-conditioned cloth dynamics prediction requires both locally plausible deformation and long-range coordination. Existing approaches largely follow two paradigms. Mesh-based GNNs capture local physical responses through material connectivity. However, their finite message-passing range limits coordination between topologically distant regions, while autoregressive rollouts tend to accumulate prediction errors. Transformer-based dynamics models capture long-range interactions through global attention, but often operate without explicit material connectivity and must learn local topological responses directly from data. We propose MeshPriorDiT, a hierarchical dynamics model that decomposes future cloth motion into a structured mesh prior and a generative residual. An action-conditioned mesh GNN first predicts multi-step vertex displacements, yielding a reference trajectory that respects material topology and grasp constraints. Conditioned on historical states, planned actions, and the mesh prior, a Residual DiT then uses conditional flow matching to jointly generate the residual motion not captured by the prior. The generated residual is further rescaled and decoded using material adjacency to coordinate corrections across neighboring vertices. We evaluate MeshPriorDiT on 15-step autoregressive rollouts across three cloth manipulation tasks. Averaged over the three tasks, MeshPriorDiT reduces average Global MSE by 43.42% relative to the GNN-Only baseline and by 75.03% relative to the DiT-DDPM baseline, while maintaining a favorable Edge-strain MSE comparable to that of GNN-Only.
Figures & tables
Figure 1: Overview of MeshPriorDiT. (a) Given the cloth mesh and planned actions, the GNN propagates motion along material connections to generate an action-conditioned mesh prior capturing topology-constrained local dynamics. (b) Conditioned on the prior trajectory, Residual DiT predicts a velocity field for residual generation. (c) The generated residual is coordinated over material neighborhoods through graph-Laplacian smoothing, scaled by γ , and composed with the mesh prior under grasp constraints. The bottom row illustrates the hierarchical decomposition. Orange dashed lines, blue solid lines, and green arrows denote the mesh prior, composed prediction, and generated residual, respectively.
Figure 2: Qualitative long-horizon rollouts of MeshPriorDiT. Each row shows six snapshots of corner-to-side folding, diagonal folding, and vertical lifting. MeshPriorDiT autoregressively predicts cloth deformation from the initial state and planned actions. Blue and gray lines denote the prediction and ground-truth boundary, while red points and orange lines indicate grasped vertices and planned actions.
Method
5-step Global
15-step Global
20-step Global
35-step Global
35-step Edge
DiT-DDPM
3.004
14.991
13.707
20.568
143.526
GNN-Only
1.314
6.615
5.830
15.460
6.975
MeshPriorDiT-Balanced
0.755
3.743
3.360
10.372
6.199
MeshPriorDiT-MinMSE
0.662
3.708
3.294
6.107
6.898
Table 1: Results on the test set. The best results are shown in bold, and the second-best results are underlined.
Figure 3: Per-step Global MSE over 15-step autoregressive TEST rollouts, averaged over 60 trajectories overall and 20 per action. Lower is better.
Figure 4: Roles of the mesh prior and Residual DiT. (a–b) Vertex errors before and after residual correction. (c–d) The residual corrects local deviations of the mesh prior. (e–f) Rendered prior and final predictions.
Figure 5: Topology-aware residual decoding and parameter sensitivity. The top row visualizes the raw generated residual and its topology-aware counterpart after material-neighborhood smoothing. The bottom row reports 15-step Global MSE and Edge-strain MSE under different residual gains γ and smoothing strengths λ . The outlined cells indicate the selected configuration.
We introduce ClothCode, a unified generative framework for garment state estimation and action-conditioned dynamics prediction in a shared discrete configuration space. A material-aligned tokenizer encodes garment configuration as residual vector-quantized patch tokens, while task-specific Masked Generative Transformers predict these tokens through parallel confidence-guided refinement. Across three garment templates, ClothCode reduces mean displacement error by 26.6% for state estimation and 37.0% for dynamics prediction relative to the strongest baselines. Dynamics prediction requires only 16.7 ms per window, compared with 556.2 ms for diffusion. Integrated into the same model-predictive controller, ClothCode improves real-world folding and flattening success from 56.7% to 78.9%. Further results and videos are available at: https://anonymous.4open.science/r/ClothCode-DE32
Yingdong Ru, Lipeng Zhuang, Shiyu Fan +3
The authors are with the University of Glasgow, Glasgow, United Kingdom.
Unified and scalable Transformers have recently achieved remarkable success in modeling diverse phenomena traditionally associated with computer graphics, such as 3D visual effects, rendering processes, and motion in videos. In this work, we take a step further by investigating whether modern Transformer techniques can tackle the challenging task of cloth simulation. To this end, we present ClothTransformer, a framework that reformulates cloth simulation as autoregressive sequence modeling in a learned latent space. Existing neural cloth simulators are largely specialized to single scenarios, intrinsically coupled to the mesh discretization, and lack robust collision handling. Our approach addresses these limitations through three contributions: (1) a unified Transformer architecture that handles diverse scenarios -- body-driven garments, robotic manipulation, and free-fall collisions -- under a single model and achieves approximately 4--9× lower error than prior state-of-the-art methods across all scenarios; (2) a scalable latent-space formulation that compresses arbitrary-resolution meshes into a fixed-size set of latent tokens, making temporal dynamics computation independent of mesh resolution; and (3) a diverse-scenario high-fidelity penetration-free dataset of ∼493.4k frames spanning all three settings, which enables a differentiable Continuous Collision Detection (CCD) module to suppress penetration artifacts. Project Page: https://yucrazing.github.io/clothtransformer/
Yu Zhang, Yidi Shao, Wenqi Ouyang +5
1S-Lab, Nanyang Technological University, Singapore · 2Feeling AI · University of Oxford +2
We present DiT-Garment to model dynamic 3D clothing over human body models in arbitrary motion. Unlike existing methods, DiT-Garment can animate garments with unseen designs and physical materials, while allowing for direct inference of deformations for any target pose. To achieve this, we leverage a 2D diffusion transformer architecture to learn 3D deformations in a 2D UV-space. As the result is non-deterministic, our generative model learns the distribution of possible outcomes. The template garment is represented as a 3D triangle mesh spatially aligned with a 3D human body model in a standardized pose. To work with different garment designs without the need of a common template or complex graph convolution operations, the diffusion transformer is conditioned on a 3D position map of the template, represented in UV-space, which allows to implicitly learn a deformation of the 3D space around the body in standard pose. Further conditioning on body motion and physical parameters allows to physically ground the model. We quantitatively and qualitatively evaluate DiT-Garment on both synthetic and real data. While only trained on synthetic simulations of automatically generated cloth designs, our method generalizes to captured and artist-made garment designs. Code and data are available for research purposes at https://dumoulina.github.io/dit-garment/.
Antoine Dumoulin, Laurence Boissieux, Joao Regateiro +2
Inria Centre at the University Grenoble Alpes · InterDigital Inc. · Inria, University of Rennes, CNRS, IRISA-UMR 6074