Transferring deformation between characters with different geometry and topology is challenging because conventional rigs encode behaviour through character-specific structures and correspondences. We present LatentReRig, an experimental framework that investigates whether pose-associated changes can instead be represented as reusable directions in a learned geometric latent space. An SDF-based variational autoencoder is coupled with two decoders: one reconstructs the implicit field, while the other predicts target vertex positions from source geometry and latent deformation conditioning. The source geometry may be neutral or already deformed. Experiments on a controlled humanoid dataset show that several poses induce coherent latent directions across identities, particularly for broad articulated motions. These signals can guide deformation of unseen characters, but explicit predictions remain less accurate for localized changes and corrective contributions. Diagnostic comparisons with repeated SDF sampling show that inter-identity distances exceed same-geometry resampling variability on average, while pose signals exhibit different margins above this baseline. The results support the presence of reusable pose-related structure and identify stable local conditioning and accurate mesh decoding as complementary requirements for improving transfer.
Figures & tables
Figure 1. High-level overview of LatentReRig. SDF samples are encoded into a variational latent representation used for implicit SDF reconstruction and semantic regularization. Relative latent variation subsequently conditions an explicit vertex decoder for mesh-space deformation prediction. Lower-level decoder inputs are omitted for clarity. Pipeline diagram connecting SDF encoding, implicit reconstruction, latent deformation conditioning, and explicit vertex prediction.
Figure 2. Effect of the conditioning strategy on cornerUpLf for the held-out character. Left: nearest-identity delta. Right: mean pose delta across identities. Nearest-identity conditioning yields a more localized and visually coherent mouth-corner deformation in this example; the global mean produces a pronounced unintended mouth opening. Both meshes are predictions. Two frontal wireframe face predictions. The left has a localized raised mouth corner; the right shows substantial mouth opening.
Figure 3. Neutral-to-pose predictions on the held-out character for two elbow poses. Each panel shows a predicted mesh reconstructed from a neutral source; neutral inputs and ground-truth targets are not displayed. The requested bending action is recognizable, while local distortion and pinching remain around the elbow. Two side views show a predicted elbow bend and a stronger elbow bend, with wireframe overlays revealing local joint distortion.
Figure 4. Combined browDwLf and browDwRt prediction on the held-out character. The bilateral brow-lowering action is visible, but the two sides are not reconstructed symmetrically. The panel shows the predicted combined state. Frontal wireframe face with both brows lowered and visible left-right asymmetry.
Figure 5. Corrective prediction for elbowBentLf on the held-out character. Grey: skinning-only input. Red: LatentReRig prediction. Blue: ground-truth skinning-plus-blendshape target. The meshes are spatially offset for visual comparison. The predicted correction partially improves the elbow volume but remains weaker than the target and exhibits residual artifacts. Three offset bent-arm meshes: red prediction, grey skinning-only source, and blue corrected ground truth.
Figure 6. Pairwise standardized Euclidean distances between deterministic neutral latent means for 20 selected identities. The diagonal corresponds to zero self-distance, while the non-zero off-diagonal structure indicates separation between character identities. Heatmap of pairwise standardized distances among neutral latent means for twenty character identities, with a zero diagonal.
Figure 7. SDF-resampling variability versus pose-conditioning signal. The dashed line marks Rcond/noise=1 ; points farther above the line exhibit a more clearly distinguishable conditioning signal. Plot comparing pose-conditioning distances with the empirical same-geometry SDF-resampling baseline.
Figure 8. Identity–pose interaction for four representative semantic poses. For each pose, the left matrix reports pairwise standardized Euclidean distances between posed identity latents, while the right matrix reports cosine similarity between the corresponding neutral-to-pose latent deltas. Four pairs of matrices compare identity distances and pose-delta cosine similarities for elbow, finger, and brow poses.
Statistic
Identity
Noise
N
190
190
Mean
27.01
8.30
Median
25.29
8.03
Std.
9.66
2.56
Min.
11.70
2.36
Max.
61.08
18.33
Table 1. Inter-identity and noise-only latent distances.
Figure 9. Static leakage in body and facial predictions on the held-out character. (a) The left-knee bend is recognizable, but the opposite leg is also deformed; the orange arrow marks an affected region that should remain static. (b) The left puff action is accompanied by unwanted movement on the opposite side of the face. Static leakage denotes predicted motion outside the region intended to move. Side and front views of a knee-bend prediction, with an orange arrow indicating the opposite leg, alongside a frontal puff prediction with deformation extending to the opposite cheek.
Automatic facial rigging across heterogeneous mesh topologies remains challenging because high-quality expression supervision is often tied to canonical templates, while deformation transfer to arbitrary meshes can introduce geometric artifacts and correspondence errors. We present TopoRig, a topology-agnostic facial rigging framework that predicts FACS-conditioned deformations directly on input mesh vertices while preserving the original topology. Starting from the ICT FaceKit expression model, we construct complementary supervision from accurate but template-biased common-topology rigs, topology-diverse but noisier transferred rigs, and targeted image-based cues for controls poorly captured by geometric transfer. TopoRig combines local surface geometry, landmark-relative semantic features, global shape context, and FACS controls to predict per-vertex displacements. We train on 3,496 generated identities using 45 non-gaze expression controls from the 53-control ICT FaceKit vocabulary. On held-out identities and unseen mesh topologies, TopoRig more faithfully reproduces the reference expression space than prior neural facial-rigging methods, while qualitative results show consistent localized deformations across diverse character geometries. Ablations demonstrate that semantic landmark features and complementary supervision improve cross-identity and cross-topology generalization. Overall, TopoRig amortizes heterogeneous and imperfect expression supervision into a single topology-preserving deformation model.
Andrew Fleet, Soroush Mehraban, Vida Adeli +2
1Queen’s University · 2Pickford AI · University of Toronto +1
Transferring articulated motion from monocular videos to rigged 3D characters is challenging due to pose ambiguity in 2D observations and morphological differences between source and target. Existing approaches often follow a reconstruct-then-retarget paradigm, tying transfer quality to intermediate 3D reconstruction and limiting applicability to categories with parametric templates. We propose MorphGS, a framework that formulates motion retargeting as a target-driven analysis-by-synthesis problem, directly optimizing target morphology and pose through image-space supervision. A rig-coupled morphology parameterization factorizes character identity from time-varying joint rotations, while dense 2D-3D correspondences and synthesized views provide complementary structural and multi-view guidance. Experiments on synthetic benchmarks and real-world videos show consistent improvements over baselines. Project page: https://xodus777.github.io/MorphGS/
Motion retargeting transfers motion across characters with different skeletal structures while preserving semantic intent and physical plausibility. Despite recent progress, two fundamental questions remain: (i) how can reliable source-motion semantics be learned without high-quality paired retargeting data, and (ii) how should retargeting be formulated when no reliable paired motion can serve as a definitive regression objective? Existing methods commonly preserve semantics by constraining predictions toward copied motions. However, such initializations entangle useful articulation cues with artifacts caused by mismatched skeletal proportions and body geometry. Moreover, directly regressing a final motion in one forward pass is restrictive because retargeting is inherently underdetermined, and the desired solution must balance semantic fidelity with target-specific physical and temporal constraints rather than match a unique paired target. Motivated by these limitations, we propose ReFM, a source-mesh-agnostic, energy-guided model that reformulates motion retargeting as progressive refinement. First, an SO(3) canonicalizer removes redundant global-orientation variations. Second, a cross-character semantic encoder, pretrained through contrastive learning, provides a character-invariant representation for both optimization guidance and semantic evaluation. ReFM then progressively refines an initialized target motion through a learned flow guided by semantic consistency, physical plausibility, temporal coherence, and minimal motion modification. The framework is compatible with different initialization strategies, including both direct motion copying and Autodesk HumanIK, an industry-standard full-body inverse-kinematics retargeting system.