Reusing a demonstrated gait can reduce the work of specifying the same motion for each biped. A video captures the timing of a step, but each robot must execute it with its own limbs and actuators. MotionSkill recovers a timed reference once from monocular video and adapts it to each target in simulation. It aligns root motion and body landmarks in a common frame and records which landmarks are available at each instant. At each source timestamp, the mapper preserves limb directions, uses the target segment lengths, and solves inverse kinematics under joint limits and closure constraints. In MuJoCo, we train a separate residual policy for each of ten bipeds on its mapped reference from one four-second walking clip. Each policy uses the measured robot state to correct joint commands during contact. From initial states held out during training, MotionSkill completed 98% of trials on the original video and 26% with the upper body occluded for one second.
Figures & tables
Fig. 1: MotionSkill overview. A walking video supplies one timed motion reference. The three poses show the direction of travel. Each of the ten simulated bipeds maps that reference to its own joint geometry and tracks it with a separately trained controller.
Fig. 2: MotionSkill framework and controller adapters. Source alignment supplies root motion, body points, and time. Mapping adapts these quantities to the target geometry. Joint references carry recorded and recovered motion into physical tracking. Travel and rhythm commands support the retained controller timing study. State feedback closes the physical loop in each case. Native limits and passive joints remain in the target model.
Study
Design
Video deployment
10 bodies, 2 inputs, 5 paired reset seeds per body
Recorded motion control
10 targets × 10 initial states × 20 s = 100 trials
Supporting analyses
184 timing trials and one original/occluded video pair
TABLE I: Evaluation design. Each video condition uses a single recovered reference and the same ten target bodies. Video results use held out initial states. The longer recorded walk and native policy trials measure stepping and timing.
Fig. 3: Stepping poses and leading ankle across ten bodies. A and B mark two recorded poses of each robot during the 20 s walking trial. The strip below each pair shows which ankle is farther forward from 4 to 6 s. Teal marks the left ankle and ochre the right. Distances give mean planar displacement over ten trials per body.
Body
Clean
Mask
Alt.
Travel (m)
G1
5/5
0/5
59
8.006
T1
5/5
0/5 ∗
58
6.465
Apollo
5/5
1/5
59
10.668
Berkeley
5/5
0/5
58–59
5.176
MicroDuck
5/5
0/5
52–58
1.141
H1
5/5
4/5
57–59
9.972
TABLE II: Physical execution on ten bipeds. Clean and Mask give completions out of five held out initial states on one video. In cells marked ∗ , mapping failed before simulation. All five planned trials count as failures. For the 20 s recorded reference, Alt. gives the range of leading foot changes and Travel gives the mean planar displacement of ten trials per body. All ten trials complete on every target.
Fig. 5: Timing analysis on retained native controllers. (a) We compare native velocity (NV), geometry with a native clock (NC), and continuous source phase (M) across 120 fixed cadence trials and two friction settings. (b) Forty changing cadence trials isolate continuous reference timing. Paired comparisons use the same initial state and native controller. Points show all five seeds with mean and sample standard deviation. (c) Standardized foot separation for G1 (seed 0). The dashed gray line is the source reference. Ochre triangles (NC) and teal circles (M) show the measured response under the native clock and continuous source phase, respectively.
Fig. 6: Video source alignment and embodiment mapping. Original and occluded inputs at t=1.2 and 2.4 s are shown with aligned motion and target joint references. Read each row from the input frame through the recovered motion to the mapped joint target for G1.
Learning from human videos offers a promising route to acquiring diverse manipulation skills. Extending this capability beyond tabletop settings to long-horizon mobile manipulation requires adapting and composing demonstrated interactions across changing scenes and robot configurations. We present Skill Assembly and Kinematic Imitation (SAKI), a framework connecting human-video skill acquisition, cross-demonstration assembly and closed-loop whole-body execution. SAKI prepares reusable object-centric skills that preserve task-critical interactions while allowing transfer paths to adapt. Given a goal and supplied task dependencies, it selects and orders skills, binds their object roles to the current scene, and carries scene estimates and robot configuration between successive skills. Whole-body kinematic imitation generates coordinated base, arm and gripper motion. During execution, persistent object estimates maintain task references across viewpoint changes, while visual feedback updates remaining trajectories. Real-robot experiments demonstrate skill reuse across layouts and the composition of independently demonstrated interactions into continuous mobile tasks, including tidying and wiping. Ablation results show that task-conditioned reference preparation substantially improves long-horizon task completion with whole-body optimisation and visual feedback held fixed. Check https://aus.bot/research/saki/ for video demos!
Yijie Lu, James Zhao, Weiming Zhi
School of Computer Science, The University of Sydney, Australia · Australian Centre for Robotics, The University of Sydney, Australia
Retargeting human kinematic reference motion onto a robot's morphology remains a formidable challenge. Existing methods often produce physical inconsistencies, such as foot sliding, self-collisions, or dynamically infeasible motions, which hinder downstream imitation learning. We propose a bilevel optimization framework that jointly adapts reference motions to a robot's morphology while training a tracking policy using reinforcement learning. To make the optimization tractable, we derive an approximate gradient for the upper-level loss. Our framework requires only a sparse set of semantic rigid-body correspondences and eliminates the need for manual tuning by identifying optimal values for a parameterization expressive enough to preserve characteristic motion across different embodiments. Moreover, by integrating retargeting directly with physics simulation, we produce physically plausible motions that facilitate robust imitation learning. We validate our method in simulation and on hardware, demonstrating challenging motions for morphologies that differ significantly from a human, including retargeting onto a quadruped.
General purpose humanoids require locomotion controllers that are multi-skill, perceptive, dynamic, and robust enough to go anywhere humans can. In this work, we present a two layer locomotion architecture: (1) a perceptive flow matching motion generator plans whole body trajectories from raw depth images while a (2) perceptive tracking policy trained with control-guided RL follows these motions. Both policies are trained on a library of terrain consistent motion clips created with dynamically optimized human data which yields both accurate velocity tracking and terrain consistent references. Our central contribution is a simple yet effective off-policy RL fine tuning loop that improves the motion generator. A structured search method is used with the generator to gather data for advantage weighted regression. This off-policy loop is much more sample efficient than on-policy residual fine tuning and improves terrain consistency on unseen geometries and skill compositions. We find that successful terrain traversals increased by up to 25 percentage points and skill selection improved by up to 80 percentage points. By using raw depth images to perceive the environment no odometry or height maps are needed, and outdoor deployment is easy. With two cameras, the policy can see terrain coming from further away and adjust its velocity regardless of the commanded speed so it can traverse the terrain. A single policy pair enables a Unitree G1 humanoid to walk, run, stand, jump on and off of boxes, and traverse stairs in outdoor environments. Project page: https://zolkin1.github.io/generate-track-improve/
Zachary Olkin, William D. Compton, Aaron D. Ames
Department of Control and Dynamical Systems at the California Institute of Technology