Track-and-Complete: Learning Humanoid Skills from a Single Failed Human Video

Sarmad Idrees; Jongeun Choi

cs.ROSep 29, 2026

Track-and-Complete: Learning Humanoid Skills from a Single Failed Human Video

Authors: Sarmad Idrees, Jongeun Choi

Organizations: School of Mechanical Engineering, Yonsei University, Seoul 03722, Korea

Abstract

Learning humanoid skills from videos typically requires a successful human demonstration, which often demands custom data collection. Although failures have traditionally been treated only as negative examples in robot learning, they can still reveal a usable trajectory prefix before the task fails, as well as the intended outcome. To leverage this information from a failed-attempt video, we propose TRACC, a pipeline that imitates the useful portion of the motion trajectory and then completes the task based on the inferred task outcome. The usable motion prefix serves as prior knowledge until the failure occurs, after which the task-completion reward guides the policy to learn the intended task goal without requiring a successful task trajectory. We evaluate our method on six in-the-wild failed human tasks from the Oops! dataset. Our experimental results demonstrate the effectiveness of the proposed approach for learning from failed attempts when no successful demonstration is available. Thus, these findings establish failed human videos as a viable source of supervision for humanoid skill learning.

Figures & tables

Method	Embodiment	Demonstration	Single video	Use of failure
VideoMimic [ 1 ]	Humanoid	Target video	Multi	–
MeshMimic [ 2 ]	Humanoid	Target video	$\checkmark$	–
HDMI [ 3 ]	Humanoid	Target video	$\checkmark$	–
OKAMI [ 4 ]	Humanoid	Target video	$\checkmark$	–
ORION [ 5 ]	Arm	Target video	$\checkmark$	–
LUCID [ 6 ]	Arm	Video corpus	Multi	–

TABLE I: Comparison of demonstration-based policy learning methods.

Fig. 2: The overall architecture of the Tracc pipeline. Human motion is reconstructed from the failed video, retargeted to the humanoid, and aligned with the simulated scene to train the prefix tracker. A VLM infers the intended goal, proposes task-reward candidates, and estimates a PoNR window. The tracker is then fine-tuned to complete the intended task.

Group	Component	Dim.	After $\tau$
Proprioception	Root position, orientation, velocity	13	Observed
	Joint position	29	Observed
	Joint velocity	29	Observed
	Previous action	29	Observed
Task state	Task object, goal, and scene state	$d_{\mathrm{task}}$	Observed
Reference	Activity, phase, release time $\tau$	3	Masked-out

TABLE II: Policy observation space.

Task	Selected reward terms and weights
Kick target	Pad approach ( $0.10$ ); latched strike ( $0.30$ ); $\times$ after the strike: two-foot contact ( $0.22$ ), pelvis height ( $0.18$ ), and stability and uprightness ( $0.15$ ); not fallen ( $0.05$ ).
Football	Foot–ball approach ( $0.12$ ); ball contact ( $0.30$ ); ball speed toward the goal ( $0.22$ ); ball–goal distance ( $0.18$ ); uprightness ( $0.14$ ); not fallen ( $0.04$ ); hovering without contact penalty ( $-0.10$ ).
Backflip	Landing-region approach ( $0.08$ ); left and right foot contact with load ( $0.15$ each); landing pelvis height ( $0.18$ ); low linear ( $0.18$ ) and angular ( $0.12$ ) velocity; uprightness ( $0.09$ ); latched inversion bonus ( $0.05$ ).
Box jump	Feet near the box top ( $0.15$ ); loaded top contact ( $0.30$ ); standing height ( $0.25$ ); pelvis over the box ( $0.10$ ); controlled standing ( $0.10$ ); uprightness ( $0.05$ ); settling ( $0.05$ ); non-top ( $-0.55$ ) and invalid-contact ( $-0.10$ ) penalties; reward set to $1$ on success.
Handstand	Hand-patch approach ( $0.12$ ); left and right palm contact ( $0.15$ each); bilateral palm contact ( $0.08$ ); $\times$ after bilateral contact: inversion progress ( $0.18$ ) and feet above the pelvis ( $0.22$ ); pelvis height ( $0.18$ ); settling ( $0.07$ ).
Log walk	Progress toward the far end of the log ( $0.45$ ); foot contact on the log top ( $0.30$ ); pelvis height above the log ( $0.15$ ); uprightness ( $0.10$ ); hovering without contact penalty ( $-0.20$ ).

TABLE III: Selected task rewards for unified policy training.

Fig. 4: Failed human demonstrations (top) and corresponding policy rollouts (bottom) for log walking, target kicking, and football kicking, ordered from top to bottom. Frames progress from left to right. Best visualized in zoomed view.

Task	Success criterion
Kick target	Foot strikes the target at $\geq 2.0$ m/s, upright $\geq 0.80$ , settled, held $2.0$ s.
Football	Ball enters the goal region, upright $\geq 0.80$ , settled, held $2.0$ s.
Backflip	Root becomes fully inverted, then both feet in the landing region and in contact, upright $\geq 0.80$ , settled, held $2.0$ s.
Box jump	Both feet on the box top, upright $\geq 0.80$ , settled, held $2.0$ s.
Handstand	Both hands in the support region and in contact, feet at least $0.40$ m above the pelvis, held $0.5$ s.
Log walk	Net displacement along the log $\geq 1.0$ m without falling.

TABLE IV: Task success criteria.

Fig. 5: Failed human demonstrations (top) and corresponding policy rollouts (bottom) for backflip (left) and handstand (right). Frames progress from left to right. Best visualized in zoomed view.

Policy	Kick target		Football		Backflip		Box jump		Handstand		Log walk
Policy	Survival (%)	Success (%)	Survival (%)	Success (%)	Survival (%)	Success (%)	Survival (%)	Success (%)	Survival (%)	Success (%)	Survival (%)	Success (%)
Task reward only	100.0	0.0	100.0	0.0	100.0	0.0	94.6	0.0	100.0	0.0	100.0	0.0
Unified policy (ours)	100.0	99.3	100.0	100.0	99.7	41.7	98.6	75.8	100.0	56.3	100.0	100.0

TABLE V: Comparison results for unified policy vs. task-reward-only policy.

Explore similar work

SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework
May 19, 2026Tianshu Wu, Xiangqi Kong, Yue Chen +5Humanoid Loco-Manipulation Human-To-Robot Transfer
1CFCS, School of Computer Science, Peking University · School of Computer Science and Engineering, Beihang University
Robots Acquire Manipulation Skills in Seconds from a Single Human Video
Jul 22, 2026Guangyan Chen, Meiling Wang, Te Cui +9Human-To-Robot Transfer Robot Systems
Beijing Institute of Technology · X SQUARE ROBOT · Tsinghua University
Robot Learning from Human Videos: A Survey
Apr 30, 2026Junyi Ma, Erhang Zhang, Haoran Yang +4Human-To-Robot Transfer Video Dataset
Shanghai Jiao Tong University, China · University of Cambridge, UK

May 19, 2026, cs.ROJ/K move · Enter open · S save

SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework

Tianshu Wu, Xiangqi Kong, Yue Chen, Qize Yu+4

1CFCS, School of Computer Science, Peking University · School of Computer Science and Engineering, Beihang University

Humanoid Loco-Manipulation Human-To-Robot Transfer Kinematic Priors Reward Functions

Open paper AskPDF arXiv

Building humanoid robots capable of generalizable whole-body loco-manipulation in the real world remains a fundamental challenge. Existing methods either rely on laborious task-specific reward engineering, rigidly replay reference motions that fail to generalize, or depend on costly teleoperation that limits scalability. While human videos capture diverse human behaviors, motion priors inferred from them are inherently imperfect, suffering from occlusion, contact artifacts, and retargeting errors that render them unsuitable for direct policy learning. To address this, we present SUGAR, a scalable data-driven framework that converts diverse human videos into deployable humanoid loco-manipulation skills, without any task-specific reward engineering or reference-motion conditioning at inference. SUGAR proceeds in three stages. First, a fully automated pipeline extracts kinematic interaction priors including human-object motion trajectories and contact labels from unstructured human videos. Second, a privileged physics-based refiner uses a unified mimic reward and progressive state pool to transform imperfect priors into physically feasible, high-fidelity skills. Third, refined skills are distilled into a hierarchical autonomous policy consisting of a command generator and a command tracker. We evaluate SUGAR on six representative loco-manipulation tasks in simulation and real-world humanoid hardware. Our method substantially outperforms reference-tracking baselines, and performance scales clearly with the amount of human video data. It also achieves zero-shot real-world transfer with reliable closed-loop execution, autonomous failure recovery, and stable long-horizon performance under external perturbations. Project Page: https://tianshuwu.github.io/sugar-humanoid/

Abstract

Explore similar work

SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework

Robots Acquire Manipulation Skills in Seconds from a Single Human Video

Robot Learning from Human Videos: A Survey