Achieving human-like behavior on humanoid robots has advanced rapidly by leveraging motion-capture demonstrations rather than hand-crafted cost functions. Adversarial imitation supplies a style signal that lets a policy fulfill a task while adhering to a desired style, but is difficult to transfer to hardware. Whole-body tracking through reinforcement learning instead deploys reliably and imitates demonstrations faithfully, but the resulting policies are conditioned on the future reference, so they require a reference trajectory at runtime and can fail when the motion differs from the training data. We train a humanoid policy directly with whole-body tracking rewards while restricting its actor to compact locomotion commands and robot observations. Reference motion supervises the reward and privileged critic, without supplying reference joint trajectories to the actor or requiring teacher distillation. The demonstrated style is thereby encoded in the policy through the tracking reward, so the robot reproduces the whole-body style under commanded velocities. Trained on a single random-walk motion and commanded with velocity profiles from held-out motions, our policy reaches at least 98% success in simulation and completes every trial on the Unitree G1. On held-out slow walking, whole-body tracking drops to 2% success and command-conditioned distillation to 56%, and both fail on the real robot. On the trained motions, the joint position error of our policy stays within 8% of whole-body tracking, and it has a higher style coverage than adversarial motion priors (AMP), which fell on the real robot without a safety crane.
Figures & tables
Fig. 1: Our policy learns the jogging style from a motion-capture clip (top) and reproduces it on the Unitree G1 (bottom). Only the base velocity of the clip (orange) is given to the policy as a command. The reference joints and body poses are used only during training.
Term
Error e
Weight w
σ
Tracking (exponential)
Anchor position
global anchor pos. error
1.0
0.3
Anchor orientation
global anchor ori. error
1.0
0.4
Body position
per-body rel. pos. error
0.7→1.0
0.3
Body orientation
per-body rel. ori. error
0.7→1.0
0.4
Body linear velocity
global body lin. vel. error
0.7→1.0
1.0
TABLE I: Reward terms. Tracking terms use the exponential kernel exp(−e2/σ2) ; penalties are quadratic. The body-tracking weights are ramped from their initial to final value during standing-start episodes (Sec. III-D ); elsewhere they take the final value.
Method
Ref. joints
Command
History
Supervision
Real robot
Teacher [ 18 ]
next step
no
no
tracking reward
free
Distill
no
yes
10 steps
Teacher actions
free
AMP [ 12 ]
no
yes
no
AMP reward
crane
Ours
no
yes
no
tracking reward
free
TABLE II: Inputs and training of the compared methods. All methods also observe proprioception. Command: planar velocity command. Real robot: free = without support. Ours is the only method trained with the tracking reward that needs neither the reference joints nor a teacher.
Motion
Method
Succ. ↑
MJPE ↓
MJVE ↓
MMD ↓
COV ↑
Random walk
AMP
100%
6.64
0.63
0.14
0.39
Teacher
100%
2.77
0.32
0.11
0.46
Distill
100%
2.95
0.34
0.11
0.46
Ours
100%
3.07
0.34
0.10
0.47
Backward
AMP
100%
5.06
0.46
0.13
0.29
Teacher
100%
2.64
0.23
0.10
0.38
TABLE III: Simulation, in distribution: each policy is trained and evaluated on the same motion. Success over 128 environments, MJPE in deg, MJVE in rad/s. Best values in bold.
Motion
Method
Succ. ↑
MJPE ↓
MJVE ↓
MMD ↓
COV ↑
Backward
AMP
100%
5.48
0.45
0.12
0.29
Teacher
98%
3.27
0.35
0.14
0.36
Distill
99%
4.21
0.43
0.18
0.32
Ours
100%
4.05
0.41
0.14
0.34
Side
AMP
100%
4.88
0.30
0.18
0.33
Teacher
100%
3.01
0.24
0.17
0.41
TABLE IV: Simulation, generalization: policies trained on the random walk and evaluated on held-out motions. Averages are over the motions with at least one success. Bold: best value among the methods with at least 90% success .
Motion
Method
Succ. ↑
MJPE ↓
MJVE ↓
MMD ↓
COV ↑
Jog
AMP 1 1 1 AMP ran on a safety crane. Without support, it fell in both of two attempts, so we used the crane to measure its tracking and style without damaging the robot. The crane only stops a fall and does not hold the robot while walking.
3/3
7.53 ± 0.12
1.23 ± 0.03
0.26 ± 0.01
0.20 ± 0.01
Teacher
3/3
3.78 ± 0.03
0.63 ± 0.00
0.16 ± 0.00
0.30 ± 0.00
Distill
3/3
4.53 ± 0.13
0.68 ± 0.01
0.17 ± 0.00
0.29 ± 0.01
Ours
3/3
4.00 ± 0.01
0.66 ± 0.00
0.16 ± 0.00
0.27 ± 0.00
Slow
AMP 1 1 footnotemark: 1
3/3
5.59 ± 0.22
0.21 ± 0.01
0.21 ± 0.02
0.31 ± 0.02
Teacher
3/3
2.77 ± 0.01
0.13 ± 0.00
0.10 ± 0.00
0.39 ± 0.00
TABLE V: Real robot, in distribution (jog and slow: per-motion policies, random walk: random-walk policy). Success out of 3 trials; values are mean ± std over the successful trials.
Motion
Method
Succ. ↑
MJPE ↓
MJVE ↓
MMD ↓
COV ↑
Backward
AMP 1 1 footnotemark: 1
3/3
5.46 ± 0.02
0.42 ± 0.00
0.13 ± 0.00
0.29 ± 0.00
Teacher
3/3
3.51 ± 0.03
0.35 ± 0.01
0.12 ± 0.00
0.42 ± 0.00
Distill
3/3
4.10 ± 0.02
0.41 ± 0.01
0.13 ± 0.00
0.41 ± 0.01
Ours
3/3
4.09 ± 0.02
0.40 ± 0.01
0.12 ± 0.00
0.40 ± 0.01
Side
AMP 1 1 footnotemark: 1
3/3
4.82 ± 0.03
0.27 ± 0.01
0.20 ± 0.01
0.34 ± 0.01
Teacher
3/3
2.88 ± 0.02
0.22 ± 0.00
0.15 ± 0.00
0.46 ± 0.00
TABLE VI: Real robot, generalization: policies trained on the random walk. Metrics are computed on the successful trials only; –: none succeeded. Averages are over the motions with at least one successful trial. Bold: best value among the methods that complete all trials.
Fig. 2: Share of held-out frames that lie inside the support of the random-walk training data, for the input of our policy (velocity command) and of the Teacher (reference joint positions and velocities). The numbers under each motion are the simulation success of ours / Teacher. The velocity command of our policy keeps 87 to 100% of the held-out frames inside the training support, which matches its success on all of them.
Fig. 3: Share of held-out frames inside the support of the training data, with the Teacher input split into reference joint positions and reference joint velocities. The reference joint positions, which our policy does not observe, are where the Teacher input leaves the training support on slow walking.
Fig. 4: Distribution of the velocity command in the training data (random walk, gray) and in each held-out motion (blue). The small forward velocities of slow and side walking, on which our policy succeeds, are covered by the random walk. Backward walking falls into the smaller negative-velocity part of the random walk.
Motion
Method
Succ. ↑
MJPE ↓
MJVE ↓
MMD ↓
COV ↑
Plain
B0
100%
3.07
0.34
0.10
0.47
B1
100%
3.09
0.35
0.10
0.47
B2
100%
3.06
0.34
0.10
0.47
B3
100%
3.17
0.35
0.10
0.47
Mirrored
B0
54%
6.14
0.80
0.20
0.40
B1
100%
3.14
0.35
0.10
0.47
TABLE VII: Simulation ablation on the random walk: B0 none, B1 mirrored training motion, B2 symmetry loss and data augmentation, B3 B1 and B2 combined, evaluated on the plain and the mirrored motion. Averages are over the motions with at least one success. Bold: best value among the methods with at least 90% success .
Motion
Method
Succ. ↑
MJPE ↓
MJVE ↓
MMD ↓
COV ↑
Plain
B0
3/3
3.68 ± 0.04
0.41 ± 0.00
0.13 ± 0.00
0.42 ± 0.00
B1
3/3
3.54 ± 0.03
0.41 ± 0.01
0.11 ± 0.00
0.49 ± 0.01
B2
3/3
3.44 ± 0.01
0.39 ± 0.00
0.11 ± 0.00
0.47 ± 0.00
B3
3/3
3.56 ± 0.01
0.39 ± 0.00
0.11 ± 0.00
0.45 ± 0.01
Mirrored
B0
0/3
–
–
–
–
B1
3/3
3.56 ± 0.01
0.41 ± 0.00
0.11 ± 0.00
0.48 ± 0.01
TABLE VIII: Real-robot ablation on the random walk: B0 none, B1 mirrored training motion, B2 symmetry loss and data augmentation, B3 B1 and B2 combined. Metrics are computed on the successful trials only; –: none succeeded. Averages are over the motions with at least one successful trial. Bold: best value among the methods that complete all trials.
Motion
Method
Succ. ↑
MJPE ↓
MJVE ↓
MMD ↓
COV ↑
Random walk
P → P
100%
3.07
0.34
0.10
0.47
S → S
100%
3.46
0.37
0.12
0.44
Backward
P → P
100%
4.05
0.41
0.14
0.34
S → S
99%
4.71
0.49
0.16
0.32
Side
P → P
100%
3.34
0.27
0.17
0.39
S → S
100%
3.99
0.33
0.19
0.38
TABLE IX: Simulation ablation, smoothed training motion. Both policies are trained on the random walk only, with the plain (P) or the smoothed (S) base trajectory, and evaluated on the random walk and on the held-out backward, side, and slow walking of the same variant. Metrics are computed on the successful environments only. Bold: best value.
Motion
Method
Succ. ↑
MJPE ↓
MJVE ↓
MMD ↓
COV ↑
Random walk
P → P
3/3
3.68 ± 0.04
0.41 ± 0.00
0.13 ± 0.00
0.42 ± 0.00
S → S
3/3
3.90 ± 0.01
0.46 ± 0.00
0.13 ± 0.00
0.45 ± 0.01
Backward
P → P
3/3
4.09 ± 0.02
0.40 ± 0.01
0.12 ± 0.00
0.40 ± 0.01
S → S
3/3
4.74 ± 0.04
0.45 ± 0.01
0.14 ± 0.01
0.36 ± 0.00
Side
P → P
3/3
3.29 ± 0.02
0.23 ± 0.00
0.15 ± 0.00
0.43 ± 0.01
S → S
3/3
3.88 ± 0.02
0.30 ± 0.01
0.16 ± 0.00
0.41 ± 0.01
TABLE X: Real-robot ablation, smoothed training motion. Both policies are trained on the random walk only, with the plain (P) or the smoothed (S) base trajectory, and evaluated on the random walk and on the held-out backward, side, and slow walking of the same variant. Metrics are computed on the successful trials only (mean ± std). Bold: best value.
Reinforcement learning has become the prevailing approach to humanoid locomotion control: policies transfer reliably from simulation to hardware and recover gracefully from disturbances. Motion quality, however, still lags behind: task-only rewards often converge to stiff, asymmetric gaits, while motion imitation methods improve appearance but become more sensitive to external disturbances because reference signals can oppose the transient poses needed to regain balance. We propose Predictive Style Matching, in which an offline predictor maps the robot's lower-body state history and velocity commands to interpretable upper-body joint and gait targets that shape the rewards during training. Because the targets are state-conditioned rather than time-indexed and the predictor is used only at training time, the deployed controller inherits the proprioceptive interface and inference cost of a task-only RL baseline. On the Unitree G1, in both simulation and hardware, PSM reduces upper-body style error by roughly an order of magnitude over task-only RL while preserving its fall-recovery rate, whereas the motion-imitation baseline attains the lowest style error but fails to recover from disturbances about five times as often.
Moscow Institute of Physics and Technology (MIPT), Dolgoprudny, Moscow Region, Russia · Innopolis University, Innopolis, Russia · Sber Robotics Center, Moscow, Russia
Despite recent advances in humanoid locomotion, controllers optimized for command tracking and robustness tend to produce mechanical gaits, whereas controllers tied to human motion data often fail to generalize to commands outside the data distribution. This work introduces a learning framework that balances these competing objectives to synthesize real-time steerable, robust, and biomimetic locomotion policies from human data. Using an in-house curated locomotion dataset covering diverse speeds and directions, we first learn a natural locomotion prior policy through a teacher-student distillation process. Specifically, we train a full-body reference-conditioned policy with Reinforcement Learning (RL), then distill it into a lightweight prior policy conditioned solely on proprioception and a planar torso-velocity steering command. Next, we fine-tune the prior policy with multi-task RL to expand command coverage and robustness beyond the data distribution, pairing a goal-conditioned task that tracks arbitrary commands with a reference-guided task that tracks the human data as an explicit style regularizer. We validate our framework on three humanoid robots: the Boston Dynamics Atlas R1, Atlas D1, and Unitree G1. Experimental results demonstrate robust performance across real-world scenarios, including direct user-controlled locomotion in indoor and outdoor environments, and integration as the locomotion layer within hierarchical control stacks. Benchmarks against Tabula Rasa RL policies trained without human data and ablation studies confirm that our framework yields a lightweight, deployable policy that reconstructs coordinated whole-body behavior from a steering command, retaining the human gait characteristics while remaining robust and fully steerable.
Mike Zhang, Dongho Kang, Kevin Bergamin +19
RAI Institute, USA · Boston Dynamics, USA · Carnegie Mellon University, USA
While velocity-commanded RL policies are effective for controlling quadrupeds, they are a limited interface for humanoids that need to perform complex whole-body motions. Furthermore, RL tracking policies for whole-body motions typically have limited information about the desired future motion, preventing them from performing controlled deviations to achieve long-term task success; such information is particularly relevant when a controller must prepare for transitions between behaviors. We address these challenges using concise motion representations that can be generated by a high-level controller, given a behavior label or text command. These representations can be fed directly into a tracking policy to provide a detailed motion reference. Our method exhibits a high degree of flexibility in performing real-world robot motions such as dancing, fighting, or walking based on conditional information. At matched capacity, our adopted stochastic motion tokenizer reconstructs motions from the Unitree G1 at a level that exceeds state-of-the-art approaches such as finite scalar quantization (FSQ) and the vector-quantized autoencoder (VQ-VAE). The resulting discrete interface conditions whole-body tracking policies that execute generated dance, fight, and walk motions lasting 16.8 s each and a 35 s fight-to-dance sequence on a physical Unitree G1.
Michael Drolet, Oleg Arenz, Jan Peters
Technical University of Darmstadt · German Research Center for AI (DFKI) · Robotics Institute Germany (RIG)