KungfuAthleteBot: learning high-dynamic humanoid motion from video with unified robust recovery

Zhongxiang Lei; Lulu Cao; Xuyang Wang; Tianyi Qian; Jinyan Liu; Xuesong Li

cs.ROOct 2, 2026

KungfuAthleteBot: learning high-dynamic humanoid motion from video with unified robust recovery

Authors: Zhongxiang Lei, Lulu Cao, Xuyang Wang, Tianyi Qian, Jinyan Liu, Xuesong Li

Abstract

Video is an abundant, inexpensive source of human motion data that is rich in extreme athletic behaviors. Making it usable for humanoid robots, however, is not a matter of simply retargeting a reconstructed trajectory: video-derived motion is physically inconsistent, devoid of actuation information, and says nothing about failure or recovery. We present KungfuAthleteBot (KAB), a framework that treats learning high-dynamic motion from video as the central problem and resolves each of these three failure modes in turn. (C1) We build the KungfuAthlete dataset from videos of national-level martial artists and introduce a physics-guided parabolic trajectory correction that removes height floating, ground penetration, and high-frequency jitter from reconstructed aerial and landing phases. (C2) Because video carries no force information, strict tracking of a reconstructed trajectory is dynamically infeasible, and error-driven initialization keeps re-launching the policy from infeasible aerial poses. We introduce physics-driven pseudo-low-kinetic-energy (LKE) sampling, our central mechanism for making such references learnable: it biases initialization towards dynamically feasible states, letting the policy discover feasible actuation patterns instead of imitating infeasible ones. (C3) Finally, we introduce a direct training paradigm in which disturbance rejection and fall recovery are learned inside the same policy that tracks the video motion, requiring no recovery reference data and no manual mode switching. On a humanoid robot, KAB learns dynamic skills from video and recovers from arbitrary falls in about 0.7 s, the fastest reported recovery for a unified policy. Ablations on the unified policy confirm the necessity of its components, supporting the view that repairing and compensating video data, rather than only collecting more of it, is what unlocks high-dynamic humanoid skills.

Figures & tables

Figure 1: Learning high-dynamic motion from video. (a) A dataset extracted from videos of national-level martial artists. (b) Video-derived motion carries no actuation information, so direct mimic-based tracking (e.g., BeyondMimic) fails on such references. Our curriculum (Sec. 4.3 ) and low-kinetic-energy sampling (Sec. 4.2 ) let the policy explore feasible actuation patterns. (c) A single policy performs fall recovery and motion tracking, and (d) rejects external disturbances.

Dataset	Body Lin Vel ( $w$ )	Body Ang Vel ( $w$ )	Frame Range
AMASS	$0.188\pm 0.165$	$0.510\pm 0.309$	$[2,8788]$
PHUMA	$0.193\pm 0.130$	$0.585\pm 0.368$	$[49,222]$
LAFAN1	$0.790\pm 0.251$	$1.962\pm 0.553$	$[3065,9171]$
KungfuAthlete (Ground)	$0.496\pm 0.324$	$1.537\pm 0.959$	$[24,15792]$
KungfuAthlete (Jump)	$0.948\pm 0.473$	$2.832\pm 1.431$	$[25,2842]$

Table 1: Kinematic statistics of motion datasets. Entries are means over the whole body, denoted by

w

in the column headings; frame range is given in frames.

Figure 2: Existing methods exhibit significant height floating when reconstructing jumping motions; prior work (e.g. PHUMA) therefore discards such data, wasting jump-related samples (top row). We instead enforce a physically consistent parabolic correction to refine the falling trajectories of jumping motions.

Figure 3: Overview of the training pipeline. Top-left: low-kinetic-energy (LKE) sampling draws video motion from dynamically feasible frames after a tracking failure. Bottom-left: the initial state $\mathbf{q}_{0}$ is drawn from a Bernoulli mixture over a reference frame and a gravity-based randomized fall state from $\mathcal{D}_{\mathrm{GRSI}}$ . Centre: reference-conditioned tracking rewards with recovery penalties (feet height, knee jerk, feet slip, root rotation). Right: the distributional actor–critic loop (FastSAC, Sec. 6 ).

Name	Expression	$\tau_{\mathrm{stageI}}$	$\tau_{\mathrm{stageII}}$	$\tau_{\mathrm{stageIII}}$
Anchor Z Pos. Deviation	$\mathbb{I}\Big(\\|\mathbf{p}^{\mathrm{ref}}_{anchor_{z}}-\mathbf{p}^{\mathrm{robot}}_{anchor_{z}}\\|_{2}>\tau_{\mathrm{pos_{anchor_{z}}}}\Big)$	0.5	0.3	0.5
Anchor Z Ori. Deviation	$\mathbb{I}\Big(\|(\mathbf{R}^{\mathrm{ref}}_{anchor})^{\top}\mathbf{g}-(\mathbf{R}^{\mathrm{robot}}_{anchor})^{\top}\mathbf{g}\|_{z}>\tau_{\mathrm{ori_{anchor}}}\Big)$	0.8	0.6	0.6
Anchor Ori. Deviation	$\mathbb{I}\Big(\|(\mathbf{R}^{\mathrm{ref}}_{anchor_{xy}})^{\top}\mathbf{g}-(\mathbf{R}^{\mathrm{robot}}_{anchor_{xy}})^{\top}\mathbf{g}\|_{z}>\tau_{\mathrm{ori}}\Big)$	/	0.25	0.6
Ankle Z Pos. Deviation	$\mathbb{I}\Big(\\|\mathbf{p}^{\mathrm{ref}}_{ankle_{z}}-\mathbf{p}^{\mathrm{robot}}_{ankle_{z}}\\|_{2}>\tau_{\mathrm{pos_{ankle_{z}}}}\Big)$	1.25	1.0	1.25
Wrist Z Pos. Deviation	$\mathbb{I}\Big(\\|\mathbf{p}^{\mathrm{ref}}_{wrist_{z}}-\mathbf{p}^{\mathrm{robot}}_{wrist_{z}}\\|_{2}>\tau_{\mathrm{pos_{wrist_{z}}}}\Big)$	0.5	0.5	0.5
Body Dof Deviation	$\mathbb{I}\Big(\exists b\in\mathcal{B}:\|\mathbf{p}^{\mathrm{ref}}_{k,b}-\mathbf{p}^{\mathrm{robot}}_{k,b}\|>\tau_{\mathrm{body}}\Big)$	$\pi/2$	$\pi/8$	$\pi/6$

Table 2: Bad-tracking termination conditions.

Parameter Item	Moderate	Aggressive
Push Lin. Vel.	$xy$ : $\pm 0.5$ , $z$ : $\pm 0.2$	$xy$ : $\pm 0.75$ , $z$ : $\pm 0.3$
Push Ang. Vel.	$xy$ : $\pm 0.52$ , $z$ : $\pm 0.78$	$xy$ : $\pm 0.785$ , $z$ : $\pm 1.17$
Rest Pos.	$/$	$\pm 0.15$
Rest Lin. Vel.	$/$	$xy$ : $\pm 0.75$ , $z$ : $\pm 0.3$
Rest Ang. Vel	$/$	$xy$ : $\pm 0.78$ , $z$ : $\pm 1.17$
Body Mass	1	[0.6,1.1]

Table 3: Configurations for noise and domain randomization.

Method	Video ref.	Aerial repair	Energy init.
KungfuBot Xie et al. (2025)	✓	✗	✗
OmniXtreme Wang et al. (2026)	✗	–	✗
KAB (ours)	✓	✓	✓

Table 4: Positioning on the video-to-robot pathway. Prior work assumes the processed reference is either learnable or safely discardable; we target the regime where it is neither. ✓ explicit support, ✗ absent, ‘–’ not applicable.

Motion ID	Method	$E_{\text{mpjpe}}$	$E_{\text{mpjve}}$	succ
78	TWIST	85.2 $\pm$ 23.8	18.9 $\pm$ 9.7	0/6
	GMT	68.3 $\pm$ 17.3	11.7 $\pm$ 4.8	0/6
	SONIC	71.9 $\pm$ 17.2	4.9 $\pm$ 4.6	0/6
	KAB	24.7 $\pm$ 12.1	5.4 $\pm$ 4.0	6/6
213	TWIST	87.4 $\pm$ 21.0	21.1 $\pm$ 9.8	0/6
	GMT	52.9 $\pm$ 12.8	9.7 $\pm$ 5.3	0/6

Table 5: Is the dataset challenging, and can KAB track it?

E_{\text{mpjpe}}

(

10^{-3}

rad),

E_{\text{mpjve}}

(

10^{-3}

rad/frame) and success rate over six trials. Motion IDs index clips in the released dataset. The same metrics are reported for KAB, to answer directly whether our own method can track these motions.

Panel of Figure 4 — Figure 4: (a) Reward curves of the three-stage training and the single-stage baseline on Motion 468. (b) Episode length under LKE and adaptive sampling in Stage II on Motion 278. LKE reaches usable tracking behaviour markedly earlier than error-driven adaptive sampling.

Method	Reference- free	Tracking & Recovery	No Get-Up Cmd.	Recovery Time (s)
HumanUP He et al. (2025b)	✓	✗	✗	6.708 $\pm$ 0.987
HoST Huang et al. (2025)	✓	✗	✗	2.055 $\pm$ 0.310
FIRM Xu et al. (2025)	✗	✗	✓	1.729 $\pm$ 0.424
BFM-Zero Li et al. (2025b)	✗	✓	✓	1.566 $\pm$ 0.065
Heracles Tao et al. (2026)	✗	✓	✓	2.471 $\pm$ 0.139
StableMimic Wu et al. (2026)	✗	✓	✓	n/r

Table 6: Comparison of fall recovery performance and capability unification. Within the methods listed, only KAB combines a reference-free recovery objective with unified tracking and recovery, and KAB attains the shortest recovery time under the protocol above.

Figure 5: Fall recovery from diverse fallen configurations. Top four rows: MuJoCo recovery sequences covering prone, supine and two twisted joint configurations with irregular limb postures. Bottom two rows: the corresponding real-world recovery on the Unitree G1.

Panel of Figure 6 — Figure 6: Training curves on Motion 1307. (a,b) Adding the recovery rewards does not change the episode length or the tracking error relative to the tracking-only baseline. (c,d) The full unified configuration matches the baseline’s episode length and tracking accuracy, at the cost of slower convergence ( $\sim$ 16k vs. $\sim$ 2.5k iterations). (e) Recovery components without mixed training: with no termination tolerance the episode length stays at 1, and with a 3 s tolerance it converges to only $\sim$ 150 of the full 500 steps, so the policy survives but never stands up and tracks. (f) Removing the height reward leaves partial tracking (300–400 steps) without recovery, while removing the root-orientation reward causes early collapse at $\sim$ 5k iterations.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Name	Expression	Weight
Stage-agnostic tracking reward (Stages I, II, III)
Body Dof Match	$\displaystyle\min\cos\left(\theta_{\mathrm{des}}-\theta\right)$	3.0
Penalty Action Rate	$\displaystyle\sum_{j}\\|a_{j}-a_{j}^{\mathrm{prev}}\\|^{2}$	-0.01
Limits DOF Position	$\displaystyle\sum_{j}\left[(\text{clip}(p_{j}-\text{upper}_{j},0))+(\text{clip}(\text{lower}_{j}-p_{j},0))\right]$	-10.0
Penalty Self Collision	$\mathbf{1}_{\text{self-collision}}$	-10.0
Stage II additional tracking reward

Appendix

Table 7: Physics-stability motion-tracking reward

r_{\text{mt}}

. Rows under “Stage-agnostic” are active in all stages; rows under “Stage II additional” are added on top of them in Stage II. The recovery terms are active whenever the recovery indicator is triggered.

Name	Expression	Weight
Motion Relative Body Position Error	$\displaystyle\exp\Bigg(-\frac{1}{\sigma^{2}}\,\frac{1}{B}\sum_{b}\\|\mathbf{p}_{b}^{\mathrm{rel}}-\hat{\mathbf{p}}_{b}\\|^{2}\Bigg)$	1.0
Motion Relative Body Orientation Error	$\displaystyle\exp\Bigg(-\frac{1}{\sigma^{2}}\,\frac{1}{B}\sum_{b}\\|\mathrm{quat\_error}(\mathbf{q}_{b}^{\mathrm{rel}},\hat{\mathbf{q}}_{b})\\|^{2}\Bigg)$	1.0
Motion Global Body Position Error	$\displaystyle\exp\Bigg(-\frac{1}{\sigma^{2}}\,\frac{1}{B}\sum_{b}\\|\mathbf{p}_{b}^{\mathrm{glo}}-\hat{\mathbf{p}}_{b}\\|^{2}\Bigg)$	0.5
Motion Global Body Orientation Error	$\displaystyle\exp\Bigg(-\frac{1}{\sigma^{2}}\,\frac{1}{B}\sum_{b}\\|\mathrm{quat\_error}(\mathbf{q}_{b}^{\mathrm{glo}},\hat{\mathbf{q}}_{b})\\|^{2}\Bigg)$	1.0
Motion Global Body Angular Velocity	$\displaystyle\exp\Bigg(-\frac{1}{\sigma^{2}}\,\frac{1}{B}\sum_{b}\\|\boldsymbol{\omega}_{b}-\hat{\boldsymbol{\omega}}_{b}\\|^{2}\Bigg)$	0.4
Motion Global Body Linear Velocity	$\displaystyle\exp\Bigg(-\frac{1}{\sigma^{2}}\,\frac{1}{B}\sum_{b}\\|\boldsymbol{l}_{b}-\hat{\boldsymbol{l}}_{b}\\|^{2}\Bigg)$	1.0

Appendix

Table 8: Motion tracking reward borrowed from BeyondMimic,

r_{\text{mt}}

Group	Term
Actor & critic	motion command (reference joint targets)
	reference body orientation (anchor frame)
	base angular velocity
	joint positions
	joint velocities
	previous action

Appendix

Table 9: Observation terms. Actor and critic share the reference-conditioned and proprioceptive terms; the critic additionally receives privileged terms that are unavailable at deployment. Noise is injected into actor observations only.

Figure 7: Motion 78: a 720° spin in the air, followed by a 360° kick.

Figure 8: Motion 1080: a kick followed by a roundhouse kick.

Explore similar work

KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skills
Jun 15, 2025Weiji Xie, Jinrui Han, Jiakun Zheng +6Full-Body Humanoid Control Whole-Body Control
Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiao Tong University · East China University of Science and Technology +2
Stubborn: A Streamlined and Unified Reinforcement Learning Framework for Robust Motion Tracking and Fall Recovery for Humanoids
Jun 11, 2026Xiao Ren, Yuhui Yang, Zongbiao Weng +2Full-Body Humanoid Control Robust Fall Detection
Southern University of Science and Technology, Shenzhen 518055, China
RecoverFormer: End-to-End Contact-Aware Recovery for Humanoid Robots
Apr 24, 2026Zihui LiuFull-Body Humanoid Control
Stanford University

Jun 15, 2025, cs.ROJ/K move · Enter open · S save

KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skills

Weiji Xie, Jinrui Han, Jiakun Zheng, Huanyu Li+5

Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiao Tong University · East China University of Science and Technology+2

Full-Body Humanoid Control Whole-Body Control Motion Imitation Robocasa Soft Actor-Critic Skills

Open paper AskPDF arXiv

Humanoid robots are promising to acquire various skills by imitating human behaviors. However, existing algorithms are only capable of tracking smooth, low-speed human motions, even with delicate reward and curriculum design. This paper presents a physics-based humanoid control framework, aiming to master highly-dynamic human behaviors such as Kungfu and dancing through multi-steps motion processing and adaptive motion tracking. For motion processing, we design a pipeline to extract, filter out, correct, and retarget motions, while ensuring compliance with physical constraints to the maximum extent. For motion imitation, we formulate a bi-level optimization problem to dynamically adjust the tracking accuracy tolerance based on the current tracking error, creating an adaptive curriculum mechanism. We further construct an asymmetric actor-critic framework for policy training. In experiments, we train whole-body control policies to imitate a set of highly-dynamic motions. Our method achieves significantly lower tracking errors than existing approaches and is successfully deployed on the Unitree G1 robot, demonstrating stable and expressive behaviors. The project page is https://kungfubot.github.io.

Abstract

Explore similar work

KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skills

Stubborn: A Streamlined and Unified Reinforcement Learning Framework for Robust Motion Tracking and Fall Recovery for Humanoids

RecoverFormer: End-to-End Contact-Aware Recovery for Humanoid Robots