Motion retargeting transfers motion across characters with different skeletal structures while preserving semantic intent and physical plausibility. Despite recent progress, two fundamental questions remain: (i) how can reliable source-motion semantics be learned without high-quality paired retargeting data, and (ii) how should retargeting be formulated when no reliable paired motion can serve as a definitive regression objective? Existing methods commonly preserve semantics by constraining predictions toward copied motions. However, such initializations entangle useful articulation cues with artifacts caused by mismatched skeletal proportions and body geometry. Moreover, directly regressing a final motion in one forward pass is restrictive because retargeting is inherently underdetermined, and the desired solution must balance semantic fidelity with target-specific physical and temporal constraints rather than match a unique paired target. Motivated by these limitations, we propose ReFM, a source-mesh-agnostic, energy-guided model that reformulates motion retargeting as progressive refinement. First, an SO(3) canonicalizer removes redundant global-orientation variations. Second, a cross-character semantic encoder, pretrained through contrastive learning, provides a character-invariant representation for both optimization guidance and semantic evaluation. ReFM then progressively refines an initialized target motion through a learned flow guided by semantic consistency, physical plausibility, temporal coherence, and minimal motion modification. The framework is compatible with different initialization strategies, including both direct motion copying and Autodesk HumanIK, an industry-standard full-body inverse-kinematics retargeting system.
Figures & tables
Figure 1: Overview of the proposed ReFM framework. The SO(3) canonicalizer first removes redundant global orientation from the source motion. A pretrained cross-character semantic encoder then extracts character-invariant representations from the source and candidate target motions, while a shape encoder provides geometric conditioning from the target mesh. Starting from an initialized motion, the semantic-aware refinement flow progressively applies local corrections that reduce a joint energy including semantic deviation, self-penetration, and temporal inconsistency.
Figure 2: Illustration of the proposed canonicalizer. (a) Motions with identical semantics but different global headings are aligned to a shared canonical orientation. (b) Intuitive geometric interpretation of the character’s facing direction, illustrated by the normal of the body-orientation plane formed by the root and shoulder joints.
Figure 3: Supervised contrastive pretraining of the cross-character semantic encoder. Cross-character positives contain the same motion performed by different characters (SM+DC), while negatives contain different motions performed by either the same character (DM+SC) or different characters (DM+DC). The encoder is pretrained and subsequently frozen in ReFM training.
Method
Pen↓
Semsim↑
Curv↓
MSE↓
Reference motions
Ground truth
0.141
–
0.654
–
Naive copy
0.140
0.999
0.657
0.051
HumanIK ( Autodesk, Inc., 2026 )
0.141
0.999
0.657
0.049
Skinned retargeting methods
STaR ( Yang et al., 2025 )
0.137
0.998
0.702
0.074
Table 1: Quantitative comparison on the Mixamo test dataset under the unified 65 -joint evaluation setting. ↓ and ↑ indicate that lower and higher values are better, respectively.
Figure 4: Qualitative comparison on the Mixamo dataset. From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM initialized from HumanIK. Red circles highlight representative regions with self-penetration or geometry-sensitive interactions.
Component
Metric
Initialization
Method
Canon.
Sem.
Flow
Pen↓
Semsim↑
Curv↓
MSE↓
Copy
w/o Canon.
✗
✓
✓
0.130
0.997
0.718
0.074
w/o Sem. Enc.
✓
✗
✓
0.129
0.992
0.729
0.074
w/o Flow
✓
✓
✗
0.137
0.998
0.689
0.058
ReFM
✓
✓
✓
0.117
0.998
0.772
0.086
HumanIK
w/o Canon.
✗
✓
✓
0.131
0.997
0.730
0.073
Table 2: Ablation of the canonicalizer, semantic component, and refinement flow under copy and HumanIK initialization. The w/o Flow variant uses one-step prediction. Lower is better for Pen , Curv , and MSE ; higher is better for Semsim .
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Positive
Negative
Cosine similarity
0.9969±0.0202
0.0859±0.2911
Motion-group embedding structure
Mean intra-class distance ↓
0.0441
Mean inter-class distance ↑
0.8768
Separation ratio ↑
0.9497
Appendix
Table 3: Evaluation of the pretrained cross-character semantic encoder on held-out motion groups. Positive pairs contain the same motion performed by different characters, while negative pairs contain different motions. The standard deviations are provided after ± .
λsem
Pen ↓
Curv ↓
MSE ↓
Semsim↑
0.1
0.118
0.788
0.093
0.996
0.2
0.119
0.783
0.095
0.997
0.5
0.117
0.772
0.091
0.996
Appendix
Table 4: Sensitivity of ReFM-HumanIK to λsem on the test dataset. All other energy weights and training configurations are fixed. Lower Pen, Curv, and MSE are better; higher Semsim is better.
Figure 5: Additional qualitative comparison for Body Jab Cross on the Mixamo ( Adobe, ) dataset at frame 1. From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM initialized from HumanIK. The complete character view illustrates the overall pose and semantic consistency of different retargeting methods.
Figure 6: Zoomed-in comparison corresponding to Fig. 5 . From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM initialized from HumanIK. The enlarged view highlights the local region where self-penetration and geometry-sensitive interactions occur.
Figure 7: Additional qualitative comparison for Body Jab Cross on the Mixamo ( Adobe, ) dataset at frame 21. From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM-HumanIK. The complete character view compares the overall retargeted configurations across different methods.
Figure 8: Zoomed-in comparison corresponding to Fig. 7 . From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM-HumanIK. The enlarged visualization makes local geometric conflicts and penetration artifacts more directly observable.
Figure 9: Additional qualitative comparison for Dribble on the Mixamo ( Adobe, ) dataset at frame 3. From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM initialized from HumanIK. The complete character view illustrates the preservation of the characteristic dribbling pose across different retargeting methods.
Figure 10: Zoomed-in comparison corresponding to Fig. 9 . From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM-HumanIK. The enlarged view emphasizes the arm–torso interaction and the self-penetration behavior of different methods.
Figure 11: Additional qualitative comparison for Dancing on the Mixamo ( Adobe, ) dataset at frame 32. From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM-HumanIK. The complete character view illustrates the preservation of the characteristic dancing pose across different retargeting methods.
Figure 12: Zoomed-in comparison corresponding to Fig. 11 . From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM-HumanIK. The enlarged view emphasizes the hand–leg interaction and the local self-penetration behavior of different methods.
Initialization
Initial Pen ↓
ReFM Pen ↓
Avg. Steps ↓
Zero-Step (%) ↑
Copy
0.092
0.078
0.61
39.0
HumanIK
0.091
0.079
0.60
40.5
Appendix
Table 5: Refinement behavior on the ScanRet dataset. We report the penetration rate before and after ReFM refinement, the average number of accepted refinement steps, and the proportion of sequences requiring zero refinement steps. Statistics are computed over the ScanRet test pairs. The low initial penetration and limited number of accepted updates indicate that ScanRet generally requires substantially less geometric correction than Mixamo, which has a penetration rate of approximately 0.140.
Figure 13: Qualitative temporal results on the ScanRet dataset. Three representative motion sequences are shown by uniformly sampled frames. We compare the paired ground-truth target animations with ReFM initialized from direct motion copying ( ReFM-Copy ) and HumanIK ( ReFM-HumanIK ). Since self-penetration is relatively infrequent in these ScanRet examples, the comparison primarily examines semantic preservation and temporal consistency.
Method
Pen↓
Semsim↑
Curv↓
MSE↓
Time (s) ↓
Direct optimization
0.131
0.994
0.718
0.085
4.3
ReFM-Copy
0.117
0.998
0.772
0.086
0.52
Appendix
Table 6: Inference-time analysis on the test dataset. Runtime is reported as the average inference time per motion. Direct optimization minimizes the same motion energy used to construct ReFM’s refinement supervision, but performs the optimization explicitly at test time. ↓ and ↑ indicate that lower and higher values are better, respectively.
Copy initialization
HumanIK initialization
Mean reduction Δˉ
0.023
0.024
Relative reduction
16%
17%
Median reduction
0.0051
0.0086
95% cluster-bootstrap CI
[0.0133,0.0312]
[0.0133,0.0347]
Appendix
Table 7: Statistical reliability of the penetration improvement on the Mixamo test set. Positive reductions favor refinement. Mean and relative reductions are reported at the precision used in Table 1 . Confidence intervals for the mean reduction are computed from unrounded per-pair measurements using cluster bootstrap resampling over target characters.
Pen ↓
Curv ↓
MSE ↓
Semsim↑
Method
seen
unseen
seen
unseen
seen
unseen
seen
unseen
Ground truth
0.154
0.093
0.621
0.771
—
—
0.978
0.963
Copy
0.154
0.090
0.628
0.762
0.046
0.067
0.999
0.998
HumanIK
0.155
0.091
0.628
0.759
0.045
0.065
0.999
0.997
STaR
0.150
0.088
0.673
0.805
0.072
0.083
0.999
0.998
MeshRet
0.148
0.089
0.676
0.792
0.104
0.095
0.989
0.937
Appendix
Table 8: Mixamo results by target-character familiarity. “Seen” denotes pairs whose target appears in training ( n=173 ), while “unseen” denotes pairs with an unseen target ( n=49 ; Kaya, Ortiz, and XBot). Penetration values across groups should be interpreted cautiously because their ground-truth penetration levels differ substantially.
Method
Overall motion quality ↓
Self-penetration handling ↓
Semantic preservation ↓
Avg. ↓
Reference motions
Naive copy
3.28 ± 0.10
2.99 ± 0.09
2.71 ± 0.08
2.99 ± 0.05
HumanIK
2.39 ± 0.09
2.66 ± 0.10
3.30 ± 0.10
2.78 ± 0.06
Skin-agnostic Retargeting Methods
SAN
4.64 ± 0.08
4.41 ± 0.10
4.63 ± 0.07
4.56 ± 0.05
R2ET
2.65 ± 0.10
2.59 ± 0.09
2.13 ± 0.08
2.46 ± 0.05
Appendix
Table 9: User-study mean ranks for five motion-retargeting alternatives. Rank 1 denotes the best result and rank 5 the worst. Avg. is the unweighted mean across the three criteria. Values are reported as mean rank ± standard error.
Figure 14: Guideline page of the user-study website. Participants are introduced to the evaluation protocol and the three ranking criteria: overall motion quality, self-penetration handling, and semantic preservation.
Figure 15: Example ranking page of the user-study website. The source motion is provided as the semantic reference, while the anonymized retargeting results are presented for participants to rank according to the specified evaluation criterion.
Retargeting human motion to humanoids requires preserving motion semantics while accounting for the robot's physical constraints. Kinematic pipelines prescribe a robot trajectory through geometric matching, without establishing whether it can be executed under the robot's dynamics. Prior physics-supervised neural methods train bidirectional full-sequence models on physics-refined kinematic trajectories, remaining tied to prescribed kinematic solutions and requiring noncausal context. We propose SMR, a streaming neural motion retargeter learned from semantic-conditioned physics experts. We convert human motion into robot-aware task-space targets for body landmark positions, orientations, and velocities. Physics experts track these targets directly in simulation under contact, dynamics, and actuator constraints. We train a single strictly causal Transformer on their executed state trajectories that complete the motion and pass semantic and physical validity checks. At each incoming frame, the student uses only current and past human observations to generate a robot motion reference for a downstream whole-body controller. Experiments demonstrate that semantic-conditioned physics experts generate motion with higher semantic fidelity and physical validity than direct optimization-based retargeting, while the resulting streaming retargeter achieves improved closed-loop performance over kinematic and physics-refined baselines under a common whole-body controller.
Chun Gu, Juntong Zhang, Jingyu Li +1
School of Data Science, Fudan University · Shanghai Innovation Institute · Logos Robotics Group
Retargeting human kinematic reference motion onto a robot's morphology remains a formidable challenge. Existing methods often produce physical inconsistencies, such as foot sliding, self-collisions, or dynamically infeasible motions, which hinder downstream imitation learning. We propose a bilevel optimization framework that jointly adapts reference motions to a robot's morphology while training a tracking policy using reinforcement learning. To make the optimization tractable, we derive an approximate gradient for the upper-level loss. Our framework requires only a sparse set of semantic rigid-body correspondences and eliminates the need for manual tuning by identifying optimal values for a parameterization expressive enough to preserve characteristic motion across different embodiments. Moreover, by integrating retargeting directly with physics simulation, we produce physically plausible motions that facilitate robust imitation learning. We validate our method in simulation and on hardware, demonstrating challenging motions for morphologies that differ significantly from a human, including retargeting onto a quadruped.
Retargeting motion across characters with varying body shapes while preserving interaction semantics, such as self-contact and near-body proximity, remains a challenging problem. While recent geometry-aware approaches address this by maintaining spatial relationships between predefined corresponding regions, their reliance on static correspondences often struggles when the target character exhibits exaggerated body proportions. In this paper, we present a geometry-aware motion retargeting framework that preserves interaction semantics by performing proximity matching over spatially adaptive anchors. Unlike prior methods with static anchor definitions, the proposed method dynamically repositions anchors to reachable regions on the target character. This is achieved via a Transformer-based anchor refinement strategy that predicts anchor displacements and constrains the translated anchors to remain on the target character geometry through differentiable soft projection. By incorporating pose-dependent spatial structures from the source character, the adapted anchors provide structurally coherent guidance for interaction-aware retargeting. Conditioned on these anchors, a graph-based autoencoder predicts target skeletal motion that preserves the spatial configuration of the source. To encourage task-aligned optimization between anchor adaptation and motion retargeting, we adopt an alternating training scheme in which each module is optimized in turn. Through extensive evaluations, we demonstrate that our method outperforms state-of-the-art approaches in preserving interaction fidelity across diverse character geometries.