Whole-body control plays a crucial role in enabling humanoid robots to execute expressive motions. Faithful execution requires preserving both the intended action and its expressive characteristics. Existing whole-body trackers typically balance joint and keypoint tracking with physical stability across diverse motions, but these objectives do not necessarily prioritize the expressive characteristics of each action. Our analysis shows that lower average positional tracking error does not guarantee better preservation of dynamic and postural motion characteristics. To address this gap, we propose a residual learning framework with compensation objectives targeting amplitude, energy, body expansion, body lean, and pelvis height. To concentrate compensation while limiting interference with the underlying tracking behavior, motion-aware gating allocates correction capacity according to body-part activity, preserving larger correction ranges in more active parts and restricting them elsewhere. For each frozen tracker, we train motion-specific residual policies and distill them into a unified policy shared across motion categories. Simulation and physical experiments on the Unitree G1 across eight motion categories demonstrate superior preservation of dynamic and postural motion characteristics compared with the corresponding baselines.
Figures & tables
Fig. 2: Residual compensation framework. (1) Human motions are retargeted, filtered, and segmented into units. (2) We train motion-specific teachers πres(i) (Stage I) and distill them into a unified student πresuni (Stage II). (3) The student augments the frozen tracker πwbt , with motion-aware gating setting correction ranges by body-part activity.
Fig. 3: Kinematic cues for amplitude, energy, body expansion, body lean, and pelvis height.
Descriptor
Definition
Unit
Amplitude
damp,u=
(ab,u)b∈B
m
aW,u=
ac,u
ak,u=
1≤t,t′≤Tumax∥zk,t−zk,t′∥2
Motion energy
denergy,u=
Tu−11t=2∑Tuj=1∑J(qt,j−qt−1,j)2
rad 2
Body expansion
dexp,u=
Tu1t=1∑Tus∈S∑k∈K∑∥pk,ts−pc,t∥2
m
Body lean
dlean,u=
Tu1t=1∑Tusinθt
–
Table I: Expressive motion descriptors computed over a motion unit u of Tu frames.
Fig. 4: Compensation on representative development motions: (a) amplitude/energy, (b) upper-body geometry, (c) body lean, and (d) pelvis height. Blue ghosts show references; solid robots show execution without (w/o Res.) and with (w/ Res.) compensation. Labels a–c link snapshots to times.
Term
Function
Weight
Amplitude
∣B∣1b∈B∑ρ(σA[eb,uA−0.02]+)
2
Energy
−(Tu−1)∣B∣1b∈B∑log(1+σE,beb,tE)
λE
Body expansion
4Tu1s∈S∑k∈K∑ρ(σXes,k,tX)
1
Body lean
Tu1ρ(σleanetL)
1
Pelvis height
Tu1ρ(σheightetH)
1
Tracking constraint
−[vutrk]+
λtrk
Table II: Reward functions and weights for residual policy training.
Fig. 5: Characteristic errors for (a) SONIC, (b) GMT, and (c) TWIST2 at P90/P70/P50. Cells show w/o Res. → w/ Res. using the units in Sec. IV-A . Colors encode relative error reduction: blue indicates improvement, gray deterioration, and white changes within ±2% . Color saturation occurs at ±40% .
Tracker
Method
MPKPE ↓ (cm)
DA↓ ( %H )
DE↓ ( 10−3 rad 2 )
DX↓ ( %H )
DL↓ (deg-eq)
DH↓ ( %H )
SONIC
w/o Res.
3.09
19.22
12.26
3.64
11.90
1.19
Track-Res
2.83
17.84
11.62
3.63
11.77
1.01
w/ Res.
3.30
14.79
10.66
2.13
10.72
0.71
GMT
w/o Res.
3.05
20.74
10.67
4.63
3.01
1.24
Track-Res
2.43
18.51
10.63
3.75
3.12
1.02
w/ Res.
3.04
17.22
9.01
3.60
1.15
0.72
Table III: We compare frozen tracking (w/o Res.), tracking residuals (Track-Res), and expressive compensation (w/ Res.).
Fig. 6: Motion-aware gating across SONIC, GMT, and TWIST2, in order within each group. (a)–(c) Applied correction (deg) versus normalized reference activity. (d)–(f) Low-activity limb endpoint error (mm; lower is better).
Fig. 7: Physical executions without (left) and with (right) compensation; blue overlays show references. Arrows in (c) show hand excursion; markers show hand separation in (d) and projected reference deviations in (e)–(g).
Tracker
Gate
DA↓ ( %H )
DEpart↓ ( 10−3 rad 2 )
MPJPE ↓ (cm)
Low limb ↓ (cm)
Corr. RMS (deg)
SONIC
Uniform
16.56
11.13
3.50
5.45
4.39
MAG
14.04
10.69
3.34
4.43
4.72
GMT
Uniform
17.09
9.73
3.31
5.03
4.40
MAG
15.03
9.37
3.01
4.26
4.86
TWIST2
Uniform
14.53
10.50
3.33
5.85
4.25
MAG
11.46
9.75
2.90
4.69
4.56
Table IV: We compare uniform and motion-aware gates (MAG) at inference with matched correction-range budgets and fixed policies.
Tracker
Method
DA↓ ( %H )
DE↓ ( 10−3 rad 2 )
DX↓ ( %H )
DL↓ (deg-eq)
DH↓ ( %H )
SONIC
w/o Res.
19.14
13.40
3.73
12.61
1.27
w/ Res.
15.73
11.46
2.97
11.93
1.05
GMT
w/o Res.
14.55
9.31
4.85
4.07
1.07
w/ Res.
10.53
8.09
4.39
2.54
0.91
TWIST2
w/o Res.
18.59
8.26
3.10
2.75
1.58
w/ Res.
16.98
7.21
2.96
2.28
1.15
Table V: We compare executions without and with residual compensation on the physical Unitree G1.
Precise end-effector tracking during humanoid whole-body motion is challenging due to floating-base oscillations, gravity, dynamic coupling, and locomotion-induced disturbances. We propose ResGAC, a whole-body humanoid controller for precise end-effector pose tracking that combines geometric admittance control (GAC) with residual reinforcement learning. GAC provides structured \SE task-space feedback and generates nominal arm joint-position targets, while residual RL compensates for unmodeled dynamics and coordinates locomotion and balance in the shared joint-position action space. The left-invariant geometric formulation allows the same GAC law to be used across manipulation reference frames. This enables the use of a ground-attached heading frame that preserves planar locomotion while removing pelvis roll, pitch, and heave from the manipulation reference, thereby reducing reference-induced end-effector motion during locomotion. ResGAC is validated on a real Unitree G1 humanoid. Across four standing end-effector tracking benchmarks, ResGAC consistently outperforms representative baselines, including SONIC, achieving lower translational and rotational errors. Real-world experiments further demonstrate reduced propagation of pelvis motion to the desired end-effector pose using the proposed ground-attached heading frame. ResGAC achieves 90% success in a standing peg-in-hole task compared with 50% for SONIC, and accurate world-frame \SE end-effector pose tracking during lower-body motion. Experimental videos are included in the supplementary material and are also available on the project website: https://resgac.github.io/ResGAC-website/.
Joohwan Seo, Xiaofeng Guo, Jinkun Cao +4
University of California, Berkeley · Amazon FAR · Carnegie Mellon University
While velocity-commanded RL policies are effective for controlling quadrupeds, they are a limited interface for humanoids that need to perform complex whole-body motions. Furthermore, RL tracking policies for whole-body motions typically have limited information about the desired future motion, preventing them from performing controlled deviations to achieve long-term task success; such information is particularly relevant when a controller must prepare for transitions between behaviors. We address these challenges using concise motion representations that can be generated by a high-level controller, given a behavior label or text command. These representations can be fed directly into a tracking policy to provide a detailed motion reference. Our method exhibits a high degree of flexibility in performing real-world robot motions such as dancing, fighting, or walking based on conditional information. At matched capacity, our adopted stochastic motion tokenizer reconstructs motions from the Unitree G1 at a level that exceeds state-of-the-art approaches such as finite scalar quantization (FSQ) and the vector-quantized autoencoder (VQ-VAE). The resulting discrete interface conditions whole-body tracking policies that execute generated dance, fight, and walk motions lasting 16.8 s each and a 35 s fight-to-dance sequence on a physical Unitree G1.
Michael Drolet, Oleg Arenz, Jan Peters
Technical University of Darmstadt · German Research Center for AI (DFKI) · Robotics Institute Germany (RIG)
We present AnyBody, a unified whole-body humanoid controller driven by an arbitrary subset of body keypoints chosen at deploy time. Prior physics-based trackers either rely on expensive full-body motion capture and error-prone trajectory retargeting, which bottleneck scalable data collection and policy learning, or decompose upper- and lower-body control into separate hierarchical representations, sacrificing the coordinated whole-body motions that loco-manipulation requires. We close this gap by learning a single latent motion representation that any keypoint subset can address. To achieve this, we first train a privileged teacher tracker on a large unstructured motion corpus and distill it online into a deterministic encoder-decoder student whose latent space is a unit sphere. We then train a transformer keypoint encoder that admits any subset of body keypoints through masked self-attention, aligning it to the privileged latent. Additionally, we treat the frozen decoder as a motor prior and specialize downstream tasks with a lightweight residual corrector in the latent space. We demonstrate the effectiveness of AnyBody by tracking large-scale human motions from arbitrary keypoint subsets, free-form control, flexibly teleoperating, and learning downstream behaviors including locomotion, in-air writing, and obstacle-reach.
Shuning Li, Sikai Li, Jiachen Li +1
University of North Carolina at Chapel Hill · Georgia Institute of Technology