Achieving real-time physics-based animation that generalizes across diverse 3D shapes and discretizations remains a fundamental challenge. We introduce PhysSkin, a physics-informed framework that addresses this challenge. In the spirit of Linear Blend Skinning, we learn continuous skinning fields as basis functions lifting motion subspace coordinates to full-space deformation, with subspace defined by handle transformations. To generate mesh-free, discretization-agnostic, and physically consistent skinning fields that generalize well across diverse 3D shapes, PhysSkin employs a new neural skinning fields autoencoder which consists of a transformer-based encoder and a cross-attention decoder. Furthermore, we also develop a novel physics-informed self-supervised learning strategy that incorporates on-the-fly skinning-field normalization and conflict-aware gradient correction, enabling effective balancing of energy minimization, spatial smoothness, and orthogonality constraints. PhysSkin shows outstanding performance on generalizable neural skinning and enables real-time physics-based animation.
Figures & tables
Figure 2: Given a static 3D shape, we first sample volumetric cubature points for animation and surface points for shape encoding. A shape encoder processes the surface points to produce shape latents, from which a set of learnable queries extract handle latents via cross-attention. Subsequently, spatial queries attend to the handle latents through another cross-attention to derive point-wise features, which are decoded into neural skinning fields. A geometry-only, physics-informed learning strategy optimizes the network in a self-supervised manner under physical and orthogonal constraints, enabling the learned skinning fields to support real-time, physics-based animation.
Figure 3: Evolution of neural skinning fields during optimization. Starting from disordered initial representations, our physics-informed self-supervised learning progressively organizes them into physically consistent, geometrically orthogonal, and spatially smooth skinning results. Each skinning weight i is scaled by max(abs(Wi)) to fall within [−1,1] and centered around 0.
Method
Ωorth×10−2↓
κlog↓
Hspec↑
RigNet [ 52 ]
0.5324
2.7997
0.9762
M-I-A [ 15 ]
1.4098
27.7357
0.7224
Anymate [ 11 ]
1.5737
2.6093
0.9682
Puppeteer [ 43 ]
0.5615
5.5605
0.9798
Ours
0.0033
1.0453
0.9999
Table 1: Quantitative comparison on RigNet [ 52 ] dataset.
Figure 4: Qualitative comparisons on RigNet [ 52 ] dataset. We visualize combined skinning fields obtained by blending all skinning weights together, providing an overall view of deformation influences and smoothness compared with baselines [ 52 , 15 , 11 , 43 ] .
Figure 5: Qualitative results on unseen 3D shapes from the RigNet [ 52 ] test set. We visualize blended skinning fields to demonstrate our method’s generalization to novel 3D shapes. All results are produced by a single unified PhysSkin model.
Figure 6: Qualitative comparisons on ShapeNet [ 5 ] dataset. We compare our methods with Simplicits [ 36 ] , Anymate [ 11 ] , and Puppeteer [ 43 ] . Zoom in for more details.
Figure 7: Qualitative skinning results of our method on unseen objects from the ShapeNet [ 5 ] dataset. Zoom in for more details.
Figure 8: Skinning results for the deformable Lego shape family, where the deformable geometries are from Mani-GS [ 14 ] .
Generalizing real-time physics-based animation across diverse 3D shapes and spatial discretizations remains challenging. In this paper, we introduce PhysSkin++, a physics-informed framework that learns continuous neural skinning fields from static geometries to construct compact deformation subspaces. Specifically, we propose a Neural Skinning Fields Autoencoder that predicts shape-dependent deformation subspaces without per-object network training. To learn these subspaces without simulation trajectories, we develop a Physics-Informed Self-Supervised Learning strategy that combines skinning-field normalization with conflict-aware gradient correction, stabilizing joint optimization under physical and orthogonality constraints. The learned representation supports efficient reduced dynamics with user control and contact interactions, together with direct animation binding to meshes, point clouds, and 3D Gaussians. For prescribed rig animation, we further introduce a self-supervised complementarity objective to learn complementary elastic responses without enforcing complementarity constraints at runtime. Experiments demonstrate cross-shape generalization of the learned skinning fields, real-time reduced simulation, and applications to controllable, collision-aware, and rig-complementary animation.
Yuanhang Lei, Baofei Liu, Xingxuan Li +9
State Key Laboratory of CAD&CG, Zhejiang University · Ant Group, Hangzhou 310000, China · Beijing Institute for General Artificial Intelligence (BIGAI) +2
Creating photorealistic, animatable 3D human avatars from monocular images still largely depends on Linear Blend Skinning (LBS) and parametric body models, which constrain expressivity and often introduce artifacts due to imperfect fitting. We propose LUNA, an LBS-free universal neural animation model that directly maps multiple 2D controls like images, keypoints, sketches, and unseen characters into 3D Gaussian deformations, bypassing explicit body fitting. At its core, a transformer-based motion regressor disentangles global rigid motion from fine-grained local dynamics to capture both coherent movement and subtle non-rigid effects. To resolve the inherent ambiguity of 2D-to-3D lifting while scaling beyond fitted datasets, we introduce hybrid supervision that distills soft structural priors from an LBS teacher and a loss that supports training on both limited fitted data and large in-the-wild unlabeled videos. Extensive experiments show LUNA achieves competitive visual fidelity compared to LBS-based approaches, while delivering realistic human motion and zero-shot cross-identity generalization across diverse driving modalities. To the best of our knowledge, LUNA is the first end-to-end 3D animatable model that supports implicit 2D driving.
Peng Li, Rawal Khirodkar, Junxuan Li +6
The Hong Kong University of Science and Technology · Work was done during an internship at Meta. · Codec Avatars Lab, Meta
Animatable 3D assets, defined as geometry equipped with an articulated skeleton and skinning weights, are fundamental to interactive graphics, embodied agents, and animation production. While recent 3D generative models can synthesize visually plausible shapes from images, the results are typically static. Obtaining usable rigs via post-hoc auto-rigging is brittle and often produces skeletons that are topologically inconsistent with the generated geometry. We present AniGen, a unified framework that directly generates animate-ready 3D assets conditioned on a single image. Our key insight is to represent shape, skeleton, and skinning as mutually consistent S3 Fields (Shape, Skeleton, Skin) defined over a shared spatial domain. To enable the robust learning of these fields, we introduce two technical innovations: (i) a confidence-decaying skeleton field that explicitly handles the geometric ambiguity of bone prediction at Voronoi boundaries, and (ii) a dual skin feature field that decouples skinning weights from specific joint counts, allowing a fixed-architecture network to predict rigs of arbitrary complexity. Built upon a two-stage flow-matching pipeline, AniGen first synthesizes a sparse structural scaffold and then generates dense geometry and articulation in a structured latent space. Extensive experiments demonstrate that AniGen substantially outperforms state-of-the-art sequential baselines in rig validity and animation quality, generalizing effectively to in-the-wild images across diverse categories including animals, humanoids, and machinery. Homepage: https://yihua7.github.io/AniGen-web/
Yi-Hua Huang, Zi-Xin Zou, Yuting He +6
The University of Hong Kong, China · VAST, China · The Chinese University of Hong Kong, China +1