HIGenNTO: Scalable Humanoid Interaction Generation via Noise-Space Trajectory Optimization

Lalit Jayanti; Kashu Yamazaki; Yuto Shibata; Kotaro Amaya; Katerina Fragkiadaki

cs.ROSep 18, 2026

HIGenNTO: Scalable Humanoid Interaction Generation via Noise-Space Trajectory Optimization

Authors: Lalit Jayanti, Kashu Yamazaki, Yuto Shibata, Kotaro Amaya, Katerina Fragkiadaki

Organizations: Carnegie Mellon University · Keio AI Research Center · Keio University

Abstract

Humanoid robots can acquire complex skills by imitating kinematic humanoid motion references, yet reliable references for contact-rich interactions remain difficult to obtain: motion capture deteriorates under occlusion and close physical contact, while retargeting introduces additional contact and geometric inconsistencies. We present HIGenNTO, a framework that synthesizes humanoid-scene interaction motion references by optimizing the initial noise of a pretrained text-conditioned motion model under sparse spatiotemporal and scene constraints. The same formulation satisfies desired contacts, avoids collisions, and maintains stable support while retaining the prior's realism and temporal coherence, generating interaction motions from scratch and composing long-horizon behaviors stage-wise. Across robot-environment and robot-object tasks, HIGenNTO produces motions that can be executed by tracking policies in simulation and used to train depth-conditioned visuomotor policies operating solely from onboard sensing. We deploy these policies on a Unitree G1 across four contact-rich tasks. Finally, the task specifications themselves can be written by a coding agent, which proposes interaction tasks and compiles them into prompt, constraint, and scene programs, authoring three of our eight evaluated tasks and four further behaviors. Together, these results establish a scalable path from high-level task descriptions to physically executable humanoid interactions.

Figures & tables

Figure 1: HIGenNTO synthesizes humanoid–scene interaction motion references through constrained optimization over a generative motion prior. Stair traversal, sitting and standing, and box pickup and placement across randomized scenes, without any motion capture of joint human–scene interaction.

Figure 2: System overview. (A) Task specification: a text prompt, sparse spatiotemporal constraints on end-effector and root goals, and a scene representation (SDF / height-field). (B) Noise-space optimization: differentiable scene and contact losses (goal, collision, foot, edge) are backpropagated through the denoising chain of a frozen motion prior into its input latent noise, yielding a scene-aware interaction dataset. (C) Long-horizon generation: temporal windows optimized in sequence, with key poses transferred across boundaries. (D) Motion repair: an inconsistent retargeted motion sequence is inverted to latent noise and re-optimized. (E) Deployment: the dataset trains a PPO tracking teacher, distilled into a depth- and proprioception-conditioned student.

Figure 3: Generated interaction motions across the task suite. Each panel overlays frames from a single generated motion, labeled in-panel: the eight tasks we evaluate, together with four further behaviors whose task programs were written by the coding agent of Sec. 4.3 .

Figure 4: Humanoid–scene interaction synthesis with HIGenNTO and baseline methods. Rows, top to bottom: HIGenNTO (ours, blue), kinematic-constraint conditioning (Kimodo, yellow), classifier guidance (Kimodo-CG, green), and best-of-ten sampling (Kimodo best-of-10, purple); each column shows the identical scene, spawn, and constraint set. Stair and step panels overlay time samples of one motion sequence (opacity increases with time); chair panels show the final seated frame, and box panels the frame after grasping.

Figure 5: Policy learning on Sit on a Chair . Task success over reinforcement-learning iterations for generated reference-based motion tracking by HIGenNTO -Teacher policies and four alternative training objectives. Each curve represents one training run.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Real-world deployment of the distilled students, using only onboard sensor observations. The robot approaches a chair and sits down ( Sit on a Chair ); stands back up from the seated state ( Standing Up ); lifts a box from the floor onto a table ( Pick and Place on Table ); and picks up and carries a box ( Pick and Place Box ).

Figure 7: Evaluation of the trained student policies in simulation. The inset in each frame is the depth image the policy receives at that instant.

Root path error (cm) $\downarrow$
	Free	Object		Terrain
Method	Slalom	Pick and	Pick and Place	Climbing	Descending	Step	Sit on	Standing
	Walking	Place Box	on Table	Stairs	Stairs	Up/Down	a Chair	Up
Ours	4.53 $\pm$ 0.82	2.65 $\pm$ 0.26	2.73 $\pm$ 0.53	2.78 $\pm$ 1.63	6.70 $\pm$ 3.40	3.89 $\pm$ 0.61	3.86 $\pm$ 0.74	2.87 $\pm$ 0.29
Kimodo	7.23 $\pm$ 3.20	2.93 $\pm$ 0.65	2.29 $\pm$ 0.51	34.00 $\pm$ 27.82	16.41 $\pm$ 7.50	2.86 $\pm$ 0.33	4.61 $\pm$ 1.15	2.96 $\pm$ 0.23
Kimodo-CG	7.23 $\pm$ 2.84	2.58 $\pm$ 0.32	2.10 $\pm$ 0.35	10.29 $\pm$ 8.85	12.81 $\pm$ 5.57	2.90 $\pm$ 0.66	4.49 $\pm$ 0.99	2.95 $\pm$ 0.26

Appendix

Table 1: Motion-generation ablation, per task, on our dataset. All four methods are provided identical scene and constraints for each sequence. Bold

=

best mean per metric. Metric definitions in Appendix C.2 .

Figure 8: Motion reference repair. HIGenNTO initialized with DDIM-inverted noise.

Joint error (rad) $\downarrow$
	Free	Object		Terrain
Method	Slalom	Pick and	Pick and Place	Climbing	Descending	Step	Sit on	Standing
	Walking	Place Box	on Table	Stairs	Stairs	Up/Down	a Chair	Up
Teacher (ours)	0.141 $\pm$ 0.014	0.182 $\pm$ 0.022	0.296 $\pm$ 0.065	0.087 $\pm$ 0.008	0.107 $\pm$ 0.011	0.099 $\pm$ 0.010	0.087 $\pm$ 0.008	0.068 $\pm$ 0.012
SONIC	0.231 $\pm$ 0.059	0.114 $\pm$ 0.009	0.100 $\pm$ 0.025	0.326 $\pm$ 0.128	0.220 $\pm$ 0.104	0.124 $\pm$ 0.050	0.115 $\pm$ 0.025	0.388 $\pm$ 0.138
Root position error (m) $\downarrow$

Appendix

Table 2: Motion tracking, per task. Bold

=

best mean per metric. Setup and rubrics in Appendix E.2 .

Figure 9: Agent-authored behaviors across their randomization settings. Each panel overlays frames from a single generated motion. The three rows are different scene configurations, and the columns are (a) Crawl Under , varying slab clearance and length; (b) Duck Under , varying slab clearance and depth; (c) Squeeze Through Gap , varying gap width; and (d) Push a Box , varying box path. The prompt and constraints behind every panel were written by the coding agent, with no task-specific demonstrations and no hand-authored constraints.

Task	$K$	$\eta$	$w_{c}$	$w_{f}$	$w_{e}$	$w_{h}$
Chair (sit, stand)	50	0.01	0.1	0.1	0	0
Stairs (up, down)	50	0.05	1.0	1.5	1.0	0
Step up/down	50	0.05	2.0	1.5	1.0	0
Slalom	100	0.05	2.0	1.5	0	0
Pick and place	100	0.01	0.5	0.1	0	1.0
Agent-authored (Appendix F.2 )

Appendix

Table 3: Optimizer settings and loss weights per task family, for the eight evaluated tasks and the four agent-authored behaviors.

K

is the number of outer Adam iterations of Alg. 1 and

\eta

its learning rate;

w_{g}{=}1.0

throughout. The two pick tasks share one recipe, as do the two chair tasks and the two stair tasks. The agent-authored tasks were all generated with a single recipe, differing only in whether the task carries a grip.

Robot Motion Generation

Robot motion generation creates feasible, task-directed trajectories for robots to move, manipulate objects, and interact with people and environments.

See all 198 papers

GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training
Jun 11, 2026Zhihai Bi, Qiang Zhang, Guoyang Zhao +8Contact-Rich Robotic Manipulation Robot Motion Generation
The Hong Kong University of Science and Technology (Guangzhou) · Artificial General Intelligence Institute, University of Science and Technology of China · The University of Hong Kong +1
SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interaction
Jun 25, 2026Sirui Chen, Shibo Zhao, Zhen Wu +3Robotic Control Humanoid Robot Locomotion
Stanford, Amazon FAR United States · Amazon FAR United States · CMU, Amazon FAR United States
HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction
Sep 28, 2026Jihwan Shin, Adrià López Escoriza, Junzhe He +2Contact-Rich Robotic Manipulation Synthetic Data Generation for Robotics
Robotic Systems Lab, ETH Zürich, 8092 Zürich, Switzerland

Jun 11, 2026J/K move · Enter open · S save

GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training

Zhihai Bi, Qiang Zhang, Guoyang Zhao, Jiahang Cao+7

The Hong Kong University of Science and Technology (Guangzhou) · Artificial General Intelligence Institute, University of Science and Technology of China · The University of Hong Kong+1

Contact-Rich Robotic Manipulation Robot Motion Generation Robot Imitation Learning Humanoid Robotics Human-Object Interaction

Open paper AskPDF arXiv

Humanoid-Object Interaction (HOI) is a fundamental capability for humanoid robots, yet it remains challenging due to the tight coupling between dynamic balance and stable interaction with diverse objects. Existing methods often require time-consuming task-specific policy training or rely on rigid trajectory replay, which limits their ability to accommodate novel interaction scenarios. In this work, we present GenHOI, a simple yet effective framework that enables humanoid robots to perform diverse object-interaction tasks in a zero-shot manner by directly imitating a single generated video, without task-specific training or physical demonstration data. GenHOI first reconstructs the robot-object scene in simulation and renders a first-frame image, which, together with the language command, conditions the synthesis of a task-oriented interaction video. The generated video is then analyzed to identify interaction-relevant contact events and estimate hand-object contact regions, which are encoded as object-centric geometric constraints that convert visual interaction cues into physically grounded optimization priors. Guided by these priors, the reference motion recovered from the video is refined and smoothed to resolve the scale ambiguity inherent in 2D video generation, while adapting a single reference trajectory to unseen robot-object relative poses. The optimized trajectory is finally executed by a closed-loop tracking controller. We validate the proposed framework in extensive simulation and real-world experiments across diverse object-interaction tasks, including box grasping, asymmetric bimanual chair carrying, table lifting from below, and cylindrical-object enveloping.

Abstract

Robot Motion Generation

Explore similar work

GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training

SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interaction

HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction