Pro-HOI: Perceptive Root-guided Humanoid-Object Interaction

Yuhang Lin; Jiyuan Shi; Dewei Wang; Jipeng Kong; Yong Liu; Chenjia Bai; Xuelong Li

cs.ROMar 1, 2026

Pro-HOI: Perceptive Root-guided Humanoid-Object Interaction

Authors: Yuhang Lin, Jiyuan Shi, Dewei Wang, Jipeng Kong, Yong Liu, Chenjia Bai, Xuelong Li

Organizations: Institute of Artificial Intelligence (TeleAI), China Telecom · Zhejiang University · University of Science and Technology of China · ShanghaiTech University

Abstract

Executing reliable Humanoid-Object Interaction (HOI) tasks for humanoid robots is hindered by the lack of generalized control interfaces and robust closed-loop perception mechanisms. In this work, we introduce Perceptive Root-guided Humanoid-Object Interaction, Pro-HOI, a generalizable framework for robust humanoid loco-manipulation. First, we collect box-carrying motions that are suitable for real-world deployment and optimize penetration artifacts through a Signed Distance Field loss. Second, we propose a novel training framework that conditions the policy on a desired root-trajectory while utilizing reference motion exclusively as a reward. This design not only eliminates the need for intricate reward tuning but also establishes root trajectory as a universal interface for high-level planners, enabling simultaneous navigation and loco-manipulation. Furthermore, to ensure operational reliability, we incorporate a persistent object estimation module. By fusing real-time detection with Digital Twin, this module allows the robot to autonomously detect slippage and trigger re-grasping maneuvers. Empirical validation on a Unitree G1 robot demonstrates that Pro-HOI significantly outperforms baselines in generalization and robustness, achieving reliable long-horizon execution in complex real-world scenarios.

Figures & tables

Method	No External Devices	Whole-Body Control	Generalizable Interaction	Controlable	Failure Recovery
OmniRetarget [ 43 ]	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$
HDMI [ 37 ]	$\times$	$\checkmark$	$\times$	$\times$	$\times$
DemoHLM	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\times$
PhysHSI [ 34 ]	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\times$
Falcon [ 47 ]	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$
OURS	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

TABLE I: Capability comparison across different methods

Fig. 2: Overview of Pro-HOI. a) Data Preparation : We frist collect human motion clips using the mocap system, and augment them with object geometries in Blender . Then we use SDF-based optimization to generate physically feasible reference motions. b) Root-Guided Policy Learning : The RL policy is trained to perform whole-body interaction skills conditioned on the desired root trajectory, while utilizing the reference motion as rewards. c) Real-world Deployment : We integrate FoundationPose for 6D object pose estimation and FAST-LIO2 [ 42 ] for root pose estimation, combined with the Interaction Pose Prior and a task specific planner to generate target root trajectories. The full stack executes entirely onboard a Jetson NX with a D435i camera and a Mid-360 LiDAR, achieving robust sim-to-real transfer.

Method	In Distribution				Out Distribution
Method	R ${}_{\text{succ}}$ (% $\uparrow$ )	R ${}_{\text{prec}}$ (cm $\downarrow$ )	R ${}_{\text{task}}$ (% $\uparrow$ )	V ${}_{\text{exec}}$ (m/s $\uparrow$ )	R ${}_{\text{succ}}$ (% $\uparrow$ )	R ${}_{\text{prec}}$ (cm $\downarrow$ )	R ${}_{\text{task}}$ (% $\uparrow$ )	V ${}_{\text{exec}}$ (m/s $\uparrow$ )
w/o SDF	97.90	7.54	55.76	0.29	82.52	8.44	64.26	0.436
w/o contact	94.99	7.86	31.13	0.29	88.43	8.28	71.65	0.41
PhysHSI [ 34 ]	96.60	7.62	90.59	0.49	82.54	7.25	70.17	0.72
Ours	97.90	6.32	73.07	0.29	91.90	6.83	76.15	0.42
Ours w/ FR.	100.00	5.80	73.67	0.29	99.93	6.84	88.38	0.27

TABLE II: Quantitative comparison of success rates, placement precision and execution velocity in Mujoco.

Method	RPE $\downarrow$	ROE $\downarrow$	MPL $\downarrow$	CPL $\downarrow$
w/o Contact	0.23 $\pm 0.03$	6.39 $\pm 0.53$	18.37 $\pm 1.54$	20.50 $\pm 7.00$
w/o SDF	0.26 $\pm 0.03$	6.33 $\pm 0.79$	22.11 $\pm 1.35$	14.25 $\pm 3.28$
Ours	0.22 $\pm 0.02$	5.77 $\pm 0.41$	16.21 $\pm 1.61$	14.80 $\pm 3.89$

TABLE III: Quantitative comparison of tracking fidelity and synchronization. Metrics are reported as Mean

\pm

Std . RPE : Root Position Error (m), ROE : Root Orientation Error (rad), MPL : Motion Phase Lag (s), CPL : Contact Phase Lag (s). The best results are highlighted in bold .

Profile	Command Velocity		GSR ( $\uparrow$ )	PP (m) ( $\downarrow$ )
Profile	$v$ (m/s)	$\omega$ (rad/s)	GSR ( $\uparrow$ )	PP (m) ( $\downarrow$ )
Slow	0.2	0.6	7 / 10	0.15
Middle	0.4	1.0	21 / 28	0.16
Fast	0.6	1.3	6 / 10	0.25

TABLE IV: Real-world performance under different dynamic command profiles. GSR : Grasp Success Rate (%), PP : Placement Precision (error in m). The velocities represent the maximum commanded linear (

v

) and angular (

\omega

) limits.

Fig. 5: The percentages quantify the likelihood of successfully detecting an object dropped within each respective region. (a) The detection probability distribution simulated via the object estimate module. (b) The baseline detection probability distribution relying exclusively on the onboard camera.

Fig. 6: The example of trajectory generation with contact information and simulated tracking contact.

Table 9

State term	Actor Dim	Critic Dim
Global root position	3	3
Global root orientation	6	6
Relative object position	3	3
Relative object orientation	6	6
Contact information	1	1
Root projected gravity	3	3

TABLE VII: Actor and critic observation state space.

Domain Randomization	Sampling Distribution
Physical parameters
Static friction coefficients	$\mu_{s}\sim\mathcal{U}[0.3,1.6]$
Dynamic friction coefficients	$\mu_{d}\sim\mathcal{U}[0.3,1.2]$
Restitution coefficient	$e\sim\mathcal{U}[0,0.5]$
Default joint positions	$\bm{q}_{0}\sim\bm{q}_{0}+\mathcal{U}[-0.01,0.01]$
Base COM offset (x, y, z)	$\Delta x\sim\mathcal{U}[-0.025,0.025],\ \Delta y\sim\mathcal{U}[-0.05,0.05],\ \Delta z\sim\mathcal{U}[-0.05,0.05]$

TABLE VIII: Domain randomization parameters applied during training.

\mathcal{U}[\cdot]

: uniform distribution.

Explore similar work

GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training
Jun 11, 2026Zhihai Bi, Qiang Zhang, Guoyang Zhao +8Contact-Rich Robotic Manipulation Robot Motion Generation
The Hong Kong University of Science and Technology (Guangzhou) · Artificial General Intelligence Institute, University of Science and Technology of China · The University of Hong Kong +1
InterReal: A Unified Physics-Based Imitation Framework for Learning Human-Object Interaction Skills
Mar 8, 2026Dayang Liang, Yuhang Lin, Xinzhe Liu +3Imitation Learning Robot Imitation Learning
Xiamen University · Zhejiang University · ShanghaiTech University +1
ViLoMan: Learning Visual-Proprioceptive Whole-Body Loco-Manipulation Skills for Humanoid Robots
Sep 16, 2026Zejie Tian, Ruibing Hou, Bingpeng Ma +2Humanoid Loco-Manipulation Humanoid Robot Control
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, China.

Jun 11, 2026J/K move · Enter open · S save

GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training

Zhihai Bi, Qiang Zhang, Guoyang Zhao, Jiahang Cao+7

The Hong Kong University of Science and Technology (Guangzhou) · Artificial General Intelligence Institute, University of Science and Technology of China · The University of Hong Kong+1

Contact-Rich Robotic Manipulation Robot Motion Generation Robot Imitation Learning Humanoid Robotics Human-Object Interaction

Open paper AskPDF arXiv

Humanoid-Object Interaction (HOI) is a fundamental capability for humanoid robots, yet it remains challenging due to the tight coupling between dynamic balance and stable interaction with diverse objects. Existing methods often require time-consuming task-specific policy training or rely on rigid trajectory replay, which limits their ability to accommodate novel interaction scenarios. In this work, we present GenHOI, a simple yet effective framework that enables humanoid robots to perform diverse object-interaction tasks in a zero-shot manner by directly imitating a single generated video, without task-specific training or physical demonstration data. GenHOI first reconstructs the robot-object scene in simulation and renders a first-frame image, which, together with the language command, conditions the synthesis of a task-oriented interaction video. The generated video is then analyzed to identify interaction-relevant contact events and estimate hand-object contact regions, which are encoded as object-centric geometric constraints that convert visual interaction cues into physically grounded optimization priors. Guided by these priors, the reference motion recovered from the video is refined and smoothed to resolve the scale ambiguity inherent in 2D video generation, while adapting a single reference trajectory to unseen robot-object relative poses. The optimized trajectory is finally executed by a closed-loop tracking controller. We validate the proposed framework in extensive simulation and real-world experiments across diverse object-interaction tasks, including box grasping, asymmetric bimanual chair carrying, table lifting from below, and cylindrical-object enveloping.

Abstract

Explore similar work

GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training

InterReal: A Unified Physics-Based Imitation Framework for Learning Human-Object Interaction Skills

ViLoMan: Learning Visual-Proprioceptive Whole-Body Loco-Manipulation Skills for Humanoid Robots