Motion Retargeting with Spatiotemporal Preferences for Humanoid Loco-Manipulation

Xiongbo Shi; Shihang Du; Zongyou Yu; Ziyu Lin; Ao Zhou; Xudong Zhang; Haodong Zhang; Junqiao Zhao

cs.ROOct 21, 2026

Motion Retargeting with Spatiotemporal Preferences for Humanoid Loco-Manipulation

Authors: Xiongbo Shi, Shihang Du, Zongyou Yu, Ziyu Lin, Ao Zhou, Xudong Zhang, Haodong Zhang, Junqiao Zhao,

Organizations: Physical Intelligence Lab, Tongji University, Shanghai, China. · Shanghai Innovation Institute, Shanghai, China. · RobotScale.AI.

Abstract

Human motion retargeting bridges the embodiment gap between humans and humanoid robots by converting human demonstrations into reference trajectories for whole-body control and loco-manipulation. For object-interaction tasks, successful execution depends on accurately preserving body--object relations at critical stages. However, uniform per-frame iteration budgets do not account for these localized accuracy requirements, limiting the computational efficiency of retargeting. To address this limitation, we present Spatiotemporal Retargeting (STR), which uses keyframes and key body parts to allocate optimization effort. Specifically, a vision-language model selects relevant motion signals and body parts, and keyframes are identified from changes in the selected signals. Based on these selections, STR assigns additional solver iterations to keyframes and higher interaction weights to the selected parts. To further improve contact fidelity, we introduce Mechanics-Aware Contact Anchoring (MCA) to refine object-surface contact positions using an approximate object-wrench model while retaining the selected surface directions. Building on the resulting references, we develop Spatiotemporal Reinforcement Learning (STRL), which combines adaptive key-stage sampling with a contact-position reward based on target proximity for downstream control. Simulation experiments on a Unitree G1 and real-world experiments on a Unitree R1 demonstrate that STR improves geometric contact coverage and continuity at lower computational cost, while the resulting references improve downstream task completion across diverse object-interaction motions.

Figures & tables

Fig. 1: Keyframes and key body parts for human-to-humanoid retargeting. The human demonstration (top) and robot reference (bottom) illustrate an object-interaction motion. Highlighted poses mark keyframes, and dashed regions indicate body parts whose relation to the object is important for contact establishment and coordinated transport. STR uses these temporal and spatial priorities to allocate solver iterations and interaction weights.

Fig. 2: Overview of STR and STRL. (a) A vision-language model selects motion signals and key body parts, and trajectory analysis identifies keyframes. (b) STR allocates additional SQP iterations to keyframes and higher interaction weights to key body parts. MCA refines object contact anchors while keeping the selected surface directions fixed. Both enter the constrained optimization to produce dense robot references. (c) STRL trains a tracking policy with adaptive key-stage sampling and a contact-position reward that encourages key body parts to track their object-surface targets during the active contact intervals.

	Penetration		Foot Skating		Contact	Part-wise Contact		Computational
	Penetration		Foot Skating		Preservation	(Geometric)		Cost
Method	Duration (%) $\downarrow$	Max Depth (cm) $\downarrow$	Duration (%) $\downarrow$	Max Vel. (cm/s) $\downarrow$	Duration (%) $\uparrow$	PECR (%) $\uparrow$	ECC (%) $\uparrow$	SQP Iterations $\downarrow$
PHC	88.22	12.52	1.19	28.99	100.00	3.41	2.32	–
GMR	100.00	12.88	0.16	11.17	100.00	1.95	1.19	–
VideoMimic	97.52	12.25	6.17	64.06	100.00	0.16	0.12	–
OmniRetarget	0.00	0.00	0.00	0.00	99.86	33.65	24.63	1471.10

TABLE I: Retargeting quality and computational cost on the ten-task G1 evaluation set. PECR and ECC quantify geometric contact coverage and continuity.

Fig. 3: Qualitative comparison of retargeted human–object interactions. STR combines keyframe budgets and body-part weights with explicit contact-position and direction targets. The comparison focuses on the body–object configurations that support the demonstrated interaction.

Fig. 4: Effect of retargeted references on grasp execution under the original RL formulation. With OmniRetarget references, the policy fails to lift the object. With STR references, it maintains hand–object contact and completes the lift. This comparison illustrates the downstream benefit of improving reference contacts before modifying policy training.

Variant	CP (%) $\uparrow$	PECR (%) $\uparrow$	ECC (%) $\uparrow$	SQP $\downarrow$
w/o spatial	99.86	34.25	24.10	352.8
w/o temporal	99.86	78.32	68.22	1455.2
w/o both	99.86	33.65	24.63	1471.1
Full (STR)	99.86	80.19	72.54	353.1

TABLE II: Ablation study of spatial and temporal preferences on the same ten retargeting tasks.

Fig. 5: Real-world execution on the Unitree R1 with policy inference on an NVIDIA Jetson Orin NX. The image sequence illustrates robot–object interaction using our retargeting and policy-learning pipeline. Hardware evaluation is reported separately from the G1 simulation experiments.

Pipeline	SR	Object RMSE		Body RMSE
	SR	Pos.	Rot.	Pos.	Rot.
	(%) $\uparrow$	(cm) $\downarrow$	( ∘ ) $\downarrow$	(cm) $\downarrow$	( ∘ ) $\downarrow$
OmniRetarget + RL	32.97	21.14	16.81	7.01	15.72
STR + RL	81.25	8.80	11.60	6.85	15.11
STR + STRL	86.56	10.00	7.74	6.82	16.90

TABLE III: Downstream performance on the same ten tasks in G1 simulation

Metric	STR + STRL
Successful trials / total trials	10 / 10
Body-Position RMSE (cm) $\downarrow$	7.16
Body-Rotation RMSE ( ∘ ) $\downarrow$	17.30

TABLE IV: Box-carrying results on the Unitree R1 with policy inference on an NVIDIA Jetson Orin NX.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Optimizer	AdamW, zero weight decay
Parallel environments	4,096
Rollout steps per environment	24
PPO epochs / mini-batches	5 / 4
Actor / critic hidden layers	512, 256, 128
Activation	ELU

Appendix

TABLE V: G1 PPO training settings and STRL parameters.

Parameter	Value
Stage decay width $\sigma_{w}$	2 frames
Minimum retained weight-increase fraction $\rho$	0.25
Selected hand / other body-vertex multiplier	4 / 2
Adjacent object-vertex multiplier	2
Budgets $B_{\mathrm{init}},B_{\mathrm{\tau}},B_{\mathrm{base}}$	50, 4, 2
Design mass $m_{d}$	2 kg

Appendix

TABLE VI: STR parameter settings for the reported G1 comparison.

Item	Configuration
Robot	R1: 26 joints (legs 12, arms 10, waist 2, head 2)
Compute	Jetson Orin NX, 16 GB
Software	Ubuntu 20.04.5; L4T 35.3.1; Linux 5.10.104-tegra; Python 3.10.21; ONNX Runtime 1.23.2 (CPU)
Training	PPO / IsaacSim; 4,096 environments; 200 Hz physics, 50 Hz control
Actor	139 inputs, 26 outputs; MLP 512–256–128, ELU; four stage-specific ONNX models
Observations	Previous actions (26), base angular velocity (3), joint positions/velocities (26 each), reference joint positions/velocities (52), relative reference orientation (6)

Appendix

TABLE VII: R1 policy and deployment settings. Rates are configured values.

Motion Retargeting

Motion retargeting transfers movement across different bodies, skeletons, or robots while preserving its intent and physical feasibility.

See all 104 papers

Sep 29, 2026

OTRetarget: Joint Robot and Object Motion Retargeting via Optimal Transport

Transferring human motion to humanoid robots requires adapting the demonstrated motion to the robot morphology while preserving interactions with the environment. This is particularly challenging for loco-manipulation tasks, where contacts with the ground and manipulated objects must remain consistent despite differences in body proportions. Yet, skeletal motion alone does not fully describe these interactions, and fixing object trajectories limits the adaptation to a new embodiment. In this paper, we introduce OTR ETARGET, a unified approach to jointly retarget robot and multi-object motion from human demonstrations. Our approach represents surface interactions through signed distances, closest surface points, and relative directions, and uses entropic optimal transport to transfer these quantities across human, robot, and object geometries. We incorporate the resulting interaction targets into a constrained inverse kinematics formulation that balances contact preservation with motion style and jointly optimizes robot and object poses at each frame. This formulation accommodates robot-object and object-object interactions without rescaling the scene or the demonstration. We validate the proposed approach on OMOMO, where it achieves a robot- object interaction Jaccard score of 87% and a depth error of 8.7 mm, compared with 28% and 29.3 mm for OmniRetarget. Finally, we demonstrate transfer to a physical G1 humanoid using whole-body policies trained with reinforcement learning on the retargeted references, across motions including two-handed box pick-and-place onto a table.

Guillaume Besset, Erwann Carn, Timothée Carecchio +5

Inria, Département d’Informatique de l’École Normale Supérieure, PSL Research University, Paris, France

Robotic Loco-Manipulation Contact-Rich Robotic Manipulation Whole-Body Robotic Manipulation Motion Retargeting Human-to-Robot Motion Imitation

Ask PDF

Sep 28, 2026

HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction

Learning from demonstration (LfD) has enabled humanoid robots to acquire diverse whole-body skills, but extending this paradigm to human-object interaction (HOI) is limited by the availability of robot-compatible interaction references. We present HOI-Retarget, a contact-centric retargeting method that transfers HOI onto a humanoid robot for large-scale motion-data generation. Its windowed trajectory optimization uses every labeled contact as a target in the object frame, balancing body tracking, foot support and smoothness under the robot's kinematic limits. The method can augment a single demonstration across object sizes, absorb contacts reconstructed from monocular video, and extend to several robots manipulating one object. We publicly release the code and the retargeted motion dataset.

Jihwan Shin, Adrià López Escoriza, Junzhe He +2

Robotic Systems Lab, ETH Zürich, 8092 Zürich, Switzerland

Contact-Rich Robotic Manipulation Synthetic Data Generation for Robotics Motion Retargeting Humanoid Robotics Human-Object Interaction

Ask PDF

Jul 4, 2026

ObjRetarget: An Object-Aware Motion Retargeting Framework with Anthropomorphic Arm Constraints and Polyhedral Hand Modeling

Learning robot dexterous manipulation from human manipulation videos requires reliably retargeting human intent to executable robot actions while maintaining stable hand-object contact, which remains a key challenge in embodied intelligence. Existing retargeting methods often ignore explicit contact modeling or rely on reinforcement learning, resulting in limited accuracy and generalization. To address this, we propose ObjRetarget, a human-to-robot motion retargeting framework for learning robot dexterous manipulation from human videos, which integrates anthropomorphic arm trajectory constraints with structured hand-object geometric modeling. For arm motion, reference trajectories extracted from human videos are used for initialization, followed by anthropomorphic constraints and redundancy-aware optimization to generate natural and accurate movements. For hand manipulation, ObjRetarget represents multi-finger contacts using polytope clusters and preserves contact structure through geometric invariants to improve stability. Experiments on real robots show that ObjRetarget improves manipulation success rates and contact stability across multiple dexterous tasks, and generalizes well to different demonstrations, object poses, and task settings.

Yuanchuan Lai, Qing Gao, Ziyan Liang +2

School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China. · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Shenzhen 518172, China. · School of Computing, University of Portsmouth, Portsmouth PO1 3HE, UK.

Contact-Rich Robotic Manipulation Motion Retargeting Robotic Manipulation Robot Imitation Learning

Ask PDF

Abstract

Motion Retargeting

Explore similar work

OTRetarget: Joint Robot and Object Motion Retargeting via Optimal Transport

HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction

ObjRetarget: An Object-Aware Motion Retargeting Framework with Anthropomorphic Arm Constraints and Polyhedral Hand Modeling