ReFM: Semantic-Aware Refinement Flow Model for Motion Retargeting

Jingxiang Qu; Lucie Taglienti; Evan Atherton

cs.CVSep 25, 2026

ReFM: Semantic-Aware Refinement Flow Model for Motion Retargeting

Authors: Jingxiang Qu, Lucie Taglienti, Evan Atherton

Organizations: Autodesk Research

Abstract

Motion retargeting transfers motion across characters with different skeletal structures while preserving semantic intent and physical plausibility. Despite recent progress, two fundamental questions remain: (i) how can reliable source-motion semantics be learned without high-quality paired retargeting data, and (ii) how should retargeting be formulated when no reliable paired motion can serve as a definitive regression objective? Existing methods commonly preserve semantics by constraining predictions toward copied motions. However, such initializations entangle useful articulation cues with artifacts caused by mismatched skeletal proportions and body geometry. Moreover, directly regressing a final motion in one forward pass is restrictive because retargeting is inherently underdetermined, and the desired solution must balance semantic fidelity with target-specific physical and temporal constraints rather than match a unique paired target. Motivated by these limitations, we propose ReFM, a source-mesh-agnostic, energy-guided model that reformulates motion retargeting as progressive refinement. First, an SO(3) canonicalizer removes redundant global-orientation variations. Second, a cross-character semantic encoder, pretrained through contrastive learning, provides a character-invariant representation for both optimization guidance and semantic evaluation. ReFM then progressively refines an initialized target motion through a learned flow guided by semantic consistency, physical plausibility, temporal coherence, and minimal motion modification. The framework is compatible with different initialization strategies, including both direct motion copying and Autodesk HumanIK, an industry-standard full-body inverse-kinematics retargeting system.

Figures & tables

Figure 1: Overview of the proposed ReFM framework. The $\mathrm{SO}(3)$ canonicalizer first removes redundant global orientation from the source motion. A pretrained cross-character semantic encoder then extracts character-invariant representations from the source and candidate target motions, while a shape encoder provides geometric conditioning from the target mesh. Starting from an initialized motion, the semantic-aware refinement flow progressively applies local corrections that reduce a joint energy including semantic deviation, self-penetration, and temporal inconsistency.

Figure 2: Illustration of the proposed canonicalizer. (a) Motions with identical semantics but different global headings are aligned to a shared canonical orientation. (b) Intuitive geometric interpretation of the character’s facing direction, illustrated by the normal of the body-orientation plane formed by the root and shoulder joints.

Figure 3: Supervised contrastive pretraining of the cross-character semantic encoder. Cross-character positives contain the same motion performed by different characters (SM+DC), while negatives contain different motions performed by either the same character (DM+SC) or different characters (DM+DC). The encoder is pretrained and subsequently frozen in ReFM training.

Method	$\mathrm{Pen}\downarrow$	$\mathrm{Sem}_{\mathrm{sim}}\uparrow$	$\mathrm{Curv}\downarrow$	$\mathrm{MSE}\downarrow$
Reference motions
Ground truth	0.141	–	0.654	–
Naive copy	0.140	0.999	0.657	0.051
HumanIK ( Autodesk, Inc., 2026 )	0.141	0.999	0.657	0.049
Skinned retargeting methods
STaR ( Yang et al., 2025 )	0.137	0.998	0.702	0.074

Table 1: Quantitative comparison on the Mixamo test dataset under the unified

65

-joint evaluation setting.

\downarrow

and

\uparrow

indicate that lower and higher values are better, respectively.

Figure 4: Qualitative comparison on the Mixamo dataset. From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM initialized from HumanIK. Red circles highlight representative regions with self-penetration or geometry-sensitive interactions.

		Component			Metric
Initialization	Method	Canon.	Sem.	Flow	$\mathrm{Pen}\downarrow$	$\mathrm{Sem}_{\mathrm{sim}}\uparrow$	$\mathrm{Curv}\downarrow$	$\mathrm{MSE}\downarrow$
Copy	w/o Canon.	✗	✓	✓	0.130	0.997	0.718	0.074
	w/o Sem. Enc.	✓	✗	✓	0.129	0.992	0.729	0.074
	w/o Flow	✓	✓	✗	0.137	0.998	0.689	0.058
	ReFM	✓	✓	✓	0.117	0.998	0.772	0.086
HumanIK	w/o Canon.	✗	✓	✓	0.131	0.997	0.730	0.073

Table 2: Ablation of the canonicalizer, semantic component, and refinement flow under copy and HumanIK initialization. The w/o Flow variant uses one-step prediction. Lower is better for

\mathrm{Pen}

\mathrm{Curv}

, and

\mathrm{MSE}

; higher is better for

\mathrm{Sem}_{\mathrm{sim}}

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Positive	Negative
Cosine similarity	$0.9969\pm 0.0202$	$0.0859\pm 0.2911$
Motion-group embedding structure
Mean intra-class distance $\downarrow$	$0.0441$
Mean inter-class distance $\uparrow$	$0.8768$
Separation ratio $\uparrow$	$0.9497$

Appendix

Table 3: Evaluation of the pretrained cross-character semantic encoder on held-out motion groups. Positive pairs contain the same motion performed by different characters, while negative pairs contain different motions. The standard deviations are provided after

\pm

$\lambda_{\mathrm{sem}}$	Pen $\downarrow$	Curv $\downarrow$	MSE $\downarrow$	$\mathrm{Sem}_{\mathrm{sim}}$ $\uparrow$
$0.1$	0.118	0.788	0.093	0.996
$0.2$	0.119	0.783	0.095	0.997
$0.5$	0.117	0.772	0.091	0.996

Appendix

Table 4: Sensitivity of ReFM-HumanIK to

\lambda_{\mathrm{sem}}

on the test dataset. All other energy weights and training configurations are fixed. Lower Pen, Curv, and MSE are better; higher

\mathrm{Sem}_{\mathrm{sim}}

is better.

Figure 5: Additional qualitative comparison for Body Jab Cross on the Mixamo ( Adobe, ) dataset at frame 1. From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM initialized from HumanIK. The complete character view illustrates the overall pose and semantic consistency of different retargeting methods.

Figure 6: Zoomed-in comparison corresponding to Fig. 5 . From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM initialized from HumanIK. The enlarged view highlights the local region where self-penetration and geometry-sensitive interactions occur.

Figure 7: Additional qualitative comparison for Body Jab Cross on the Mixamo ( Adobe, ) dataset at frame 21. From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM-HumanIK. The complete character view compares the overall retargeted configurations across different methods.

Figure 8: Zoomed-in comparison corresponding to Fig. 7 . From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM-HumanIK. The enlarged visualization makes local geometric conflicts and penetration artifacts more directly observable.

Figure 9: Additional qualitative comparison for Dribble on the Mixamo ( Adobe, ) dataset at frame 3. From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM initialized from HumanIK. The complete character view illustrates the preservation of the characteristic dribbling pose across different retargeting methods.

Figure 10: Zoomed-in comparison corresponding to Fig. 9 . From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM-HumanIK. The enlarged view emphasizes the arm–torso interaction and the self-penetration behavior of different methods.

Figure 11: Additional qualitative comparison for Dancing on the Mixamo ( Adobe, ) dataset at frame 32. From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM-HumanIK. The complete character view illustrates the preservation of the characteristic dancing pose across different retargeting methods.

Figure 12: Zoomed-in comparison corresponding to Fig. 11 . From left to right: (a) source motion, (b) ground truth, (c) Naive copy, (d) HumanIK ( Autodesk, Inc., 2026 ) , (e) MeshRet ( Ye et al., 2024 ) , (f) SAN ( Aberman et al., 2020 ) , (g) STaR ( Yang et al., 2025 ) , (h) R2ET ( Zhang et al., 2023 ) , and (i) ReFM-HumanIK. The enlarged view emphasizes the hand–leg interaction and the local self-penetration behavior of different methods.

Initialization	Initial Pen $\downarrow$	ReFM Pen $\downarrow$	Avg. Steps $\downarrow$	Zero-Step (%) $\uparrow$
Copy	0.092	0.078	0.61	39.0
HumanIK	0.091	0.079	0.60	40.5

Appendix

Table 5: Refinement behavior on the ScanRet dataset. We report the penetration rate before and after ReFM refinement, the average number of accepted refinement steps, and the proportion of sequences requiring zero refinement steps. Statistics are computed over the ScanRet test pairs. The low initial penetration and limited number of accepted updates indicate that ScanRet generally requires substantially less geometric correction than Mixamo, which has a penetration rate of approximately 0.140.

Figure 13: Qualitative temporal results on the ScanRet dataset. Three representative motion sequences are shown by uniformly sampled frames. We compare the paired ground-truth target animations with ReFM initialized from direct motion copying ( ReFM-Copy ) and HumanIK ( ReFM-HumanIK ). Since self-penetration is relatively infrequent in these ScanRet examples, the comparison primarily examines semantic preservation and temporal consistency.

Method	$\mathrm{Pen}\downarrow$	$\mathrm{Sem}_{\mathrm{sim}}\uparrow$	$\mathrm{Curv}\downarrow$	$\mathrm{MSE}\downarrow$	Time (s) $\downarrow$
Direct optimization	0.131	0.994	0.718	0.085	4.3
ReFM-Copy	0.117	0.998	0.772	0.086	0.52

Appendix

Table 6: Inference-time analysis on the test dataset. Runtime is reported as the average inference time per motion. Direct optimization minimizes the same motion energy used to construct ReFM’s refinement supervision, but performs the optimization explicitly at test time.

\downarrow

and

\uparrow

indicate that lower and higher values are better, respectively.

	Copy initialization	HumanIK initialization
Mean reduction $\bar{\Delta}$	$0.023$	$0.024$
Relative reduction	$16\%$	$17\%$
Median reduction	$0.0051$	$0.0086$
$95\%$ cluster-bootstrap CI	$[0.0133,\,0.0312]$	$[0.0133,\,0.0347]$

Appendix

Table 7: Statistical reliability of the penetration improvement on the Mixamo test set. Positive reductions favor refinement. Mean and relative reductions are reported at the precision used in Table 1 . Confidence intervals for the mean reduction are computed from unrounded per-pair measurements using cluster bootstrap resampling over target characters.

	Pen $\downarrow$		Curv $\downarrow$		MSE $\downarrow$		$\mathrm{Sem}_{\mathrm{sim}}$ $\uparrow$
Method	seen	unseen	seen	unseen	seen	unseen	seen	unseen
Ground truth	$0.154$	$0.093$	$0.621$	$0.771$	—	—	$0.978$	$0.963$
Copy	$0.154$	$0.090$	$0.628$	$0.762$	$0.046$	$0.067$	$0.999$	$0.998$
HumanIK	$0.155$	$0.091$	$0.628$	$0.759$	$0.045$	$0.065$	$0.999$	$0.997$
STaR	$0.150$	$0.088$	$0.673$	$0.805$	$0.072$	$0.083$	$0.999$	$0.998$
MeshRet	$0.148$	$0.089$	$0.676$	$0.792$	$0.104$	$0.095$	$0.989$	$0.937$

Appendix

Table 8: Mixamo results by target-character familiarity. “Seen” denotes pairs whose target appears in training (

n=173

), while “unseen” denotes pairs with an unseen target (

n=49

; Kaya, Ortiz, and XBot). Penetration values across groups should be interpreted cautiously because their ground-truth penetration levels differ substantially.

Reference motions
Method	Overall motion quality $\downarrow$	Self-penetration handling $\downarrow$	Semantic preservation $\downarrow$	Avg. $\downarrow$
Naive copy	3.28 $\pm$ 0.10	2.99 $\pm$ 0.09	2.71 $\pm$ 0.08	2.99 $\pm$ 0.05
HumanIK	2.39 $\pm$ 0.09	2.66 $\pm$ 0.10	3.30 $\pm$ 0.10	2.78 $\pm$ 0.06
Skin-agnostic Retargeting Methods
SAN	4.64 $\pm$ 0.08	4.41 $\pm$ 0.10	4.63 $\pm$ 0.07	4.56 $\pm$ 0.05
R2ET	2.65 $\pm$ 0.10	2.59 $\pm$ 0.09	2.13 $\pm$ 0.08	2.46 $\pm$ 0.05

Appendix

Table 9: User-study mean ranks for five motion-retargeting alternatives. Rank

1

denotes the best result and rank

5

the worst. Avg. is the unweighted mean across the three criteria. Values are reported as mean rank

\pm

standard error.

Figure 14: Guideline page of the user-study website. Participants are introduced to the evaluation protocol and the three ranking criteria: overall motion quality, self-penetration handling, and semantic preservation.

Figure 15: Example ranking page of the user-study website. The source motion is provided as the semantic reference, while the anonymized retargeting results are presented for participants to rank according to the specified evaluation criterion.

Motion Retargeting

Motion retargeting transfers movement across different bodies, skeletons, or robots while preserving its intent and physical feasibility.

See all 102 papers

SMR: Learning streaming motion retargeting for humanoids from human semantic
Oct 21, 2026Chun Gu, Juntong Zhang, Jingyu Li +1Motion Retargeting Human-to-Robot Motion Imitation
School of Data Science, Fudan University · Shanghai Innovation Institute · Logos Robotics Group
ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting
May 7, 2026David Müller, Agon Serifi, Sammy Christen +3Cross-Embodiment Robot Learning Motion Retargeting
Disney Research, Switzerland
Skinned Motion Retargeting with Spatially Adaptive Interaction Guidance
May 19, 2026Soojin Choi, Seokhyeon Hong, Chaelin Kim +3Motion Retargeting
Visual Media Lab, KAIST, Republic of Korea

Oct 21, 2026J/K move · Enter open · S save

SMR: Learning streaming motion retargeting for humanoids from human semantic

Chun Gu, Juntong Zhang, Jingyu Li, Li Zhang

School of Data Science, Fudan University · Shanghai Innovation Institute · Logos Robotics Group

Motion Retargeting Human-to-Robot Motion Imitation

Open paper AskPDF arXiv

Retargeting human motion to humanoids requires preserving motion semantics while accounting for the robot's physical constraints. Kinematic pipelines prescribe a robot trajectory through geometric matching, without establishing whether it can be executed under the robot's dynamics. Prior physics-supervised neural methods train bidirectional full-sequence models on physics-refined kinematic trajectories, remaining tied to prescribed kinematic solutions and requiring noncausal context. We propose SMR, a streaming neural motion retargeter learned from semantic-conditioned physics experts. We convert human motion into robot-aware task-space targets for body landmark positions, orientations, and velocities. Physics experts track these targets directly in simulation under contact, dynamics, and actuator constraints. We train a single strictly causal Transformer on their executed state trajectories that complete the motion and pass semantic and physical validity checks. At each incoming frame, the student uses only current and past human observations to generate a robot motion reference for a downstream whole-body controller. Experiments demonstrate that semantic-conditioned physics experts generate motion with higher semantic fidelity and physical validity than direct optimization-based retargeting, while the resulting streaming retargeter achieves improved closed-loop performance over kinematic and physics-refined baselines under a common whole-body controller.

Abstract

Motion Retargeting

Explore similar work

SMR: Learning streaming motion retargeting for humanoids from human semantic

ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting

Skinned Motion Retargeting with Spatially Adaptive Interaction Guidance