KinoGen: Customizable Humanoid Loco-Manipulation References via Kinematically Grounded Video Generation

Qi Wu; Lulu Zhao; Sanghyun Hahn; Rishi Yennu; Emmy Yang; Dingqi Zhang; Tao Pang; Jiuguang Wang

cs.ROOct 14, 2026

KinoGen: Customizable Humanoid Loco-Manipulation References via Kinematically Grounded Video Generation

Authors: Qi Wu, Lulu Zhao, Sanghyun Hahn, Rishi Yennu, Emmy Yang, Dingqi Zhang, Tao Pang, Jiuguang Wang,

Organizations: Cornell University · Multiply Labs

Abstract

We present KinoGen, a framework that synthesizes customizable humanoid loco-manipulation references through kinematically grounded video generation. KinoGen conditions a pretrained video generator on calibrated renders of the target robot and object in a simulated task instance and recovers the reference by fitting the same models to the generated two-view video, so that the reference lies in the robot configuration space by construction without a human body proxy or retargeting. Body, hand, and object priors initialize the reconstruction, and direct optimization against cross-view evidence refines it. Since an edited keyframe is simply another rendered state, a user can prescribe a goal object pose, regenerate the video around the edited keyframe, and obtain a reference that satisfies the pose exactly. Across eight loco-manipulation tasks, KinoGen yields lower penetration and foot skating than video-based baselines and preserves over 90% of annotated contacts, and its customized references satisfy every prescribed target with substantially less penetration than direct interpolation. Tracking policies trained on the synthesized references are deployed on a Unitree G1 humanoid with dexterous hands. Project webpage: https://kinogen-humanoid.github.io/.

Figures & tables

Fig. 1: KinoGen synthesizes and customizes humanoid loco-manipulation references for the target embodiment. A task specification and calibrated simulator render condition robot-video generation. Fitting the target robot and object models to the generated video recovers synchronized whole-body, hand, and object trajectories. Editing a robot–object keyframe conditions regeneration of the intervening motion, while constrained reconstruction enforces the correction. The resulting references train tracking policies deployed on a Unitree G1 with dexterous hands.

Fig. 2: Overview of KinoGen. Left: A task prompt and calibrated renders of the initial robot–object state condition synchronized multi-view video generation (Sec. III-A ). Middle: Motion priors are reconciled with the generated observations to initialize reconstruction (Sec. III-B ); direct optimization then fits the robot and object configurations to multi-view evidence (Sec. III-C ). Right: Reference correction enforces prescribed robot–object configurations through keyframe editing, keyframe-conditioned regeneration, and constrained reconstruction (Sec. III-D ).

Fig. 3: Synthesized robot–object references. We show box lifting, dumbbell lifting, floor mopping, stool sitting, kneeling and reaching, and basket lifting, all produced by the same pipeline. Each panel contains temporally ordered keyframes from one generated and reconstructed trajectory.

	Reconstruction quality				Downstream RL Policy
Method	Penetration (cm) $\downarrow$	Foot skating (cm/s) $\downarrow$	Contact preservation (%) $\uparrow$	Smoothness (m/s 2 ) $\downarrow$	Success (%) $\uparrow$	Reference deviation (m) $\downarrow$	Learning curve AUC $\uparrow$
GenMimic [ 22 ]	3.17	9.95	13.57	0.96	31	0.16	29.92
GenHOI [ 1 ]	10.55	6.52	43.91	1.34	11	0.15	19.90
GRAIL [ 31 ]	3.06	5.11	69.40	0.59	42	0.13	38.73
KinoGen (ours)	2.30	4.09	91.16	0.73	57	0.09	60.74

TABLE I: Quantitative comparison of generated humanoid reference quality and downstream RL performances

Variant	Worst IoU $\uparrow$	EPE (px) $\downarrow$	Acceleration (m/s 2 ) $\downarrow$	Penetration (mm) $\downarrow$
Single view	0.42	13.80	0.47	10.4
w/o point tracks	0.65	9.89	0.39	10.0
w/o body prior	0.52	11.59	1.01	26.8
w/o hand prior	0.62	8.02	0.79	24.3
Full model	0.65	7.92	0.37	10.5

TABLE II: Ablation of calibrated visual evidence and motion priors. Results are averaged across all evaluation sequences. Each sequence is repeated 10 times.

Fig. 6: Metric reference correction. Left: Keyframe-conditioned regeneration proposes a lift-over motion, and constrained reconstruction enforces the target exactly. Middle: Reconstructing the original cue-conditioned video does not reach the prescribed target. Right: Direct end-effector interpolation reaches the target but moves the hand and bottle through the table.

	Position error (cm) $\downarrow$			Worst penetration (mm) $\downarrow$
Variant	Left	Center	Right	Left	Center	Right
w/o correction	14.55	1.07	3.06	44.84	5.14	13.55
EE interpolation	0.00	0.00	0.00	139.51	147.23	139.45
Full model	0.00	0.00	0.00	19.32	2.72	20.99

TABLE III: Metric reference correction. We report the mean target error and the mean per-trial maximum table penetration, taking the worse value between the bottle and operating hand.

Humanoid Loco-Manipulation

Humanoid loco-manipulation studies how robots coordinate walking, balance, and whole-body interaction to move and handle objects.

See all 93 papers

Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation
Sep 29, 2026Zihan Wang, Zhen Wu, Pieter Abbeel +6HOI Generation Humanoid Loco-Manipulation
Amazon FAR · UC Berkeley · Stanford +1
VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes
Jun 29, 2026Yen-Jen Wang, Jiaman Li, Sirui Chen +9Humanoid Loco-Manipulation Synthetic Data Generation for Robotics
Amazon FAR · UC Berkeley · Stanford University +1
Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers
Sep 9, 2026Tianyue Wu, Boyuan An, Shuqi Zhao +6Goal-Conditioned Robotic Manipulation Hand-Object Interaction Detection
Sharpa Robotics · The University of Hong Kong · University of California, Berkeley

Sep 29, 2026J/K move · Enter open · S save

Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation

Zihan Wang, Zhen Wu, Pieter Abbeel, Rocky Duan+5

Amazon FAR · UC Berkeley · Stanford+1

HOI Generation Humanoid Loco-Manipulation Generalization in Robotic Manipulation Synthetic Data Generation for Robotics Robot Imitation Learning

Open paper AskPDF arXiv

Teaching humanoids loco-manipulation skills, such as carrying diverse objects, via visual imitation is a promising path toward generalist robots. However, collecting diverse, high-quality interaction videos, such as clips that clearly show a person's full body and unoccluded interactions with objects, poses a practical barrier to scaling this approach. We propose PRISM, a real-to-sim-to-real framework that overcomes this limitation by amplifying a handful of real videos into a large, diverse training set. PRISM first generates hundreds of diverse "counterfactual" human-object interaction videos via video-to-video (V2V) generation from a few exemplar real videos. Our contact-anchored real-to-sim pipeline then reconstructs both human and object motions, retargeting this imperfect video data into physically plausible trajectories. The intra-class variability across these counterfactual videos lets us train a single policy that generalizes to unseen objects within each category. We demonstrate the full pipeline by deploying this policy on a real robot without any real-world fine-tuning. Using only onboard depth observations, our humanoid picks up, carries, and drops objects, including boxes, barrels, bins, and balls, across novel instances, sizes, and initial configurations.

Abstract

Humanoid Loco-Manipulation

Explore similar work

Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation

VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers