PhysSkin++: Real-Time and Generalizable Physics-Based Animation via Self-Supervised Neural Skinning
Authors: Yuanhang Lei, Baofei Liu, Xingxuan Li, Tao Cheng, Zixuan Wang, Boming Zhao, Yujun Shen, Siyuan Huang, Ruizhen Hu, Peter Yichen Chen, Hujun Bao, Zhaopeng Cui
Organizations: State Key Laboratory of CAD&CG, Zhejiang University · Ant Group, Hangzhou 310000, China · Beijing Institute for General Artificial Intelligence (BIGAI) · Shenzhen University · University of British Columbia
Generalizing real-time physics-based animation across diverse 3D shapes and spatial discretizations remains challenging. In this paper, we introduce PhysSkin++, a physics-informed framework that learns continuous neural skinning fields from static geometries to construct compact deformation subspaces. Specifically, we propose a Neural Skinning Fields Autoencoder that predicts shape-dependent deformation subspaces without per-object network training. To learn these subspaces without simulation trajectories, we develop a Physics-Informed Self-Supervised Learning strategy that combines skinning-field normalization with conflict-aware gradient correction, stabilizing joint optimization under physical and orthogonality constraints. The learned representation supports efficient reduced dynamics with user control and contact interactions, together with direct animation binding to meshes, point clouds, and 3D Gaussians. For prescribed rig animation, we further introduce a self-supervised complementarity objective to learn complementary elastic responses without enforcing complementarity constraints at runtime. Experiments demonstrate cross-shape generalization of the learned skinning fields, real-time reduced simulation, and applications to controllable, collision-aware, and rig-complementary animation.
Figures & tables
Fig. 1: PhysSkin++ is a generalizable physics-informed neural skinning framework for object animation. The framework is learned directly from static 3D geometries via physics-informed self-supervision without any annotated data. Once trained, PhysSkin++ can be applied in a feed-forward manner to perform neural skinning for diverse 3D shapes and discretizations, enabling real-time physics-based animation. It further supports interactive control, collision-aware multi-object interaction, and rig-complementary animation.
Fig. 2: Given a static 3D shape, we first sample volumetric cubature points for animation and surface points for shape encoding. A shape encoder processes the surface points to produce shape latents, from which a set of learnable queries extract handle latents via cross-attention. Subsequently, spatial queries attend to the handle latents through another cross-attention block to derive point-wise features, which are decoded into neural skinning fields. A geometry-only, physics-informed learning strategy optimizes the network in a self-supervised manner under physical and orthogonality constraints, enabling the learned skinning fields to support real-time, physics-based animation.
Fig. 3: Evolution of neural skinning fields during optimization. Starting from disordered initial representations, our physics-informed self-supervised learning progressively organizes them into physically consistent, approximately orthogonal, and spatially smooth skinning results. For visualization, each skinning weight function Wi is divided by max∣Wi∣ to fall within [−1,1] .
Fig. 4: Qualitative results for controllable and collision-aware reduced dynamics. (a) Dirichlet boundary conditions constrain prescribed regions during deformation. (b) Interactive dragging provides localized user control. (c) Boundary-penalty energies support contact with planar environments across diverse shapes. We encourage readers to visit our project page for additional animation videos.
Fig. 5: Qualitative results for collision-aware multi-object animation. (a) Rigid–deformable contact between objects with distinct geometries. (b) Deformable–deformable contact among multiple objects, jointly resolved by the reduced collision formulation. We encourage readers to visit our project page for additional animation videos.
Fig. 6: Self-supervised rig-complementarity training. The rig Jacobian of a prescribed rig provides a complementarity constraint for learning rig-complementary neural skinning fields from static geometry.
Fig. 7: Qualitative results of rig-complementary animation. For each frame, the rig-driven primary motion without complementary elastic deformation is shown semi-transparently as a reference to the corresponding rig-complementary animation. (a) During interactive rig control, user-manipulated poses define the primary animation, and PhysSkin++ produces the corresponding complementary elastic response at runtime. (b) For user-authored rig animation sequences, PhysSkin++ augments the input kinematic trajectory with temporally evolving complementary elastic deformation. We encourage readers to visit our project page for additional animation videos.
Fig. 8: Qualitative comparisons on the RigNet [ 62 ] dataset. We visualize combined skinning fields obtained by blending all skinning weights together, providing an overall view of deformation influences and smoothness compared with baselines [ 62 , 20 , 14 , 52 ] .
Fig. 9: Qualitative results on unseen 3D shapes from the RigNet [ 62 ] test set. We visualize blended skinning fields to demonstrate our method’s generalization to novel 3D shapes. A single PhysSkin++ model produces all results.
Fig. 10: Qualitative comparisons on the ShapeNet [ 7 ] dataset. We compare with Simplicits [ 44 ] , Anymate [ 14 ] , and Puppeteer [ 52 ] . Zoom in for details.
Fig. 12: Skinning results for the deformable Lego shape family, where the deformable geometries are from Mani-GS [ 18 ] .
Fig. 13: Qualitative results of self-supervised rig-complementarity learning. For each character, the four columns show the prescribed rig structure, its LBS skinning weights, and neural skinning fields learned with and without the rig-space complementarity term Lcomp , respectively. The comparison illustrates how this term changes the spatial distribution of the learned skinning fields across diverse characters and rig structures.
Fig. 14: Top: Trained without Lpot . Bottom: Trained with Lpot .
Achieving real-time physics-based animation that generalizes across diverse 3D shapes and discretizations remains a fundamental challenge. We introduce PhysSkin, a physics-informed framework that addresses this challenge. In the spirit of Linear Blend Skinning, we learn continuous skinning fields as basis functions lifting motion subspace coordinates to full-space deformation, with subspace defined by handle transformations. To generate mesh-free, discretization-agnostic, and physically consistent skinning fields that generalize well across diverse 3D shapes, PhysSkin employs a new neural skinning fields autoencoder which consists of a transformer-based encoder and a cross-attention decoder. Furthermore, we also develop a novel physics-informed self-supervised learning strategy that incorporates on-the-fly skinning-field normalization and conflict-aware gradient correction, enabling effective balancing of energy minimization, spatial smoothness, and orthogonality constraints. PhysSkin shows outstanding performance on generalizable neural skinning and enables real-time physics-based animation.
Yuanhang Lei, Tao Cheng, Xingxuan Li +6
State Key Laboratory of CAD&CG, Zhejiang University · BIGAI · Shenzhen University +1
We propose Skelebones, a Scaffold-Skin Rigging System built on three steps: (1) Bones compress temporally consistent Gaussian or mesh sequences into free-form bones with smooth skinning weights, approximating non-rigid deformations via linear blend skinning (LBS); (2) Skeleton extracts the Mean Curvature Skeleton (MCS) from the canonical shape and temporally refines its topology and kinematics into a compact skeletal structure; and (3) Binding connects the skeleton and bones through non-parametric Partwise Motion Matching (PartMM), which synthesizes novel bone motions by matching, retrieving, and blending existing ones. Together, these steps compress the dynamics of 4D shapes into compact skelebones that are simultaneously controllable and expressive. The resulting representation is category-agnostic, meaning template-free; motion-adaptive, with dynamic topology; and topology-correct, with a skeleton consistent with the surface geometry. PartMM requires no learning. We validate our method on both synthetic and real-world datasets, achieving substantial reanimation improvements on unseen poses: a 17.3 dB PSNR gain over LBS on DNA-Rendering and a 45.6 dB gain over Bag-of-Bones on ActorsHQ, while preserving high rendering fidelity for characters with complex non-rigid dynamics. PartMM generalizes robustly to both Gaussian and mesh representations, excelling in low-data regimes of approximately 1,000 frames, with a 48.4 RMSE improvement over LBS and improvements of more than 20 over GRU- and MLP-based methods. Code will be publicly released at https://cookmaker.cn/gaussianimate/.
Jiaxin Wang, Dongxin Lyu, Zeyu Cai +4
Westlake University, Hangzhou, China · Nanjing University, Nanjing, China · The University of Hong Kong, Hong Kong, China +1
3D Gaussian Splatting has achieved remarkable success in photorealistic and efficient rendering, leading to a rapid increase in 3D assets represented by 3D Gaussian primitives. Directly rigging these assets with arbitrary skeleton topologies is highly desirable. However, training a feed-forward skinning framework is infeasible due to the lack of high-quality 3D Gaussian rigging datasets. An alternative solution is to transfer mesh-based techniques to 3D Gaussian-based representation, but 3D Gaussian primitives are not restricted to the surface and lack explicit topological connectivity. Moreover, this kind of method suffers from poor generalization to unseen data due to its strong dependence on training data, while acquiring high-quality rigging data is prohibitively expensive. To address this challenging problem, we propose G-Skin, a novel generative skinning framework designed for expressive and high-fidelity animation with 3D Gaussian representation. To overcome this 3D data scarcity, we introduce a skeleton-controllable image generation model leveraging 2D vision foundation models to distill powerful motion priors into pseudo-guidance. Guided by these priors, we formulate an optimization pipeline incorporating geometry-aware regularizations, which stabilizes the learning process and ensures smooth, structurally coherent skinning weights. G-Skin also generalizes flexibly to the augmented variants of 3D Gaussian representation designed to mitigate animation-induced rendering artifacts. Extensive experiments validate the effectiveness of our approach, demonstrating clear advantages over state-of-the-art methods. Project page: https://yaoyx689.github.io/GSkin.html.
Yuxin Yao, Kendong Liu, Shiqi Zhou +2
Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · Central Media Technology Institute, Huawei, China · School of Computer Science and Engineering, Central South University, Changsha, China