Interaction is one of the core abilities of humanoid robots. However, most existing frameworks focus on non-interactive whole-body control, which limits their practical applicability. In this work, we develop InterReal, a unified physics-based imitation learning framework for Real-world human-object Interaction (HOI) control. InterReal enables humanoid robots to track HOI reference motions, facilitating the learning of fine-grained interactive skills and their deployment in real-world settings. Within this framework, we first introduce a HOI motion data augmentation scheme with hand-object contact constraints, and utilize the augmented motions to improve policy stability under object perturbations. Second, we propose an automatic reward learner to address the challenge of large-scale reward shaping. A meta-policy guided by critical tracking error metrics explores and allocates reward signals to the low-level reinforcement learning objective, which enables more effective learning of interactive policies. Experiments on HOI tasks of box-picking and box-pushing demonstrate that InterReal achieves the best tracking accuracy and the highest task success rate compared to recent baselines. Furthermore, we validate the framework on the real-world robot Unitree G1, which demonstrates its practical effectiveness and robustness beyond simulation.
Figures & tables
Fig. 2: Overall framework of InterReal. InterReal consists of three main components: motion data preprocessing, multi-motion multi-environment learning, and deployment. It enables retargeting HOI motions into trainable motions with G1 robot shapes, achieves accurate motion tracking in complex HOI training settings, and ultimately supports real-world deployment.
Box-picking task
Tracking Errors
Empjpe↓
Emllpe↓
Emulpe↓
Em3lpe↓
Emope↓
Emorae↓
Emige ( e−7 ) ↓
Emlrae↓
ASAP*
0.1634
0.0052
0.0039
0.0034
0.0087
0.0049
19.09
0.4926
InterMimic*
0.1984
0.0053
0.0043
0.0031
0.0032
0.0049
17.08
0.4873
InterReal (ours)
0.1076
0.0037
0.0028
0.0019
0.0021
0.0033
15.70
0.2893
Box-pushing task
ASAP*
0.0783
0.0015
0.0032
0.0020
0.0033
0.0014
21.41
0.1671
TABLE I: Comparison of the best mean tracking accuracy on the Box-picking and Box-pushing task. The data records the average of 20 evaluations of the best model on each tracking metric.
Fig. 3: Comparison of tracking accuracy among InterReal and baselines on the box-picking task.
Fig. 4: Ablation results for the internal coefficient δ of the meta-learning on the box-picking task.
Tasks
Box-picking (%) ↑
Box-pushing (%) ↑
ASAP*
77.38
70.63
InterMimic*
84.72
79.10
InterReal (ours)
96.41
87.45
TABLE II: Comparison results of task success rate metrics on box-picking and box-pushing tasks.
Fig. 5: Adaptive curves for reward-related weight coefficients.
Humanoid-Object Interaction (HOI) is a fundamental capability for humanoid robots, yet it remains challenging due to the tight coupling between dynamic balance and stable interaction with diverse objects. Existing methods often require time-consuming task-specific policy training or rely on rigid trajectory replay, which limits their ability to accommodate novel interaction scenarios. In this work, we present GenHOI, a simple yet effective framework that enables humanoid robots to perform diverse object-interaction tasks in a zero-shot manner by directly imitating a single generated video, without task-specific training or physical demonstration data. GenHOI first reconstructs the robot-object scene in simulation and renders a first-frame image, which, together with the language command, conditions the synthesis of a task-oriented interaction video. The generated video is then analyzed to identify interaction-relevant contact events and estimate hand-object contact regions, which are encoded as object-centric geometric constraints that convert visual interaction cues into physically grounded optimization priors. Guided by these priors, the reference motion recovered from the video is refined and smoothed to resolve the scale ambiguity inherent in 2D video generation, while adapting a single reference trajectory to unseen robot-object relative poses. The optimized trajectory is finally executed by a closed-loop tracking controller. We validate the proposed framework in extensive simulation and real-world experiments across diverse object-interaction tasks, including box grasping, asymmetric bimanual chair carrying, table lifting from below, and cylindrical-object enveloping.
Zhihai Bi, Qiang Zhang, Guoyang Zhao +8
The Hong Kong University of Science and Technology (Guangzhou) · Artificial General Intelligence Institute, University of Science and Technology of China · The University of Hong Kong +1
Whole-body Humanoid-Object Interaction (HOI) is bottlenecked by the scarcity of high-fidelity 3D data. While video generative priors offer a promising alternative, existing methods suffer from Representation Misalignment due to their reliance on geometric priors (e.g., explicit CAD models), and Retargeting Complexity arising from intensive morphing and morphological mismatch. We propose Imagine2Real, a zero-shot HOI framework for flexible, geometry-free interaction. To resolve misalignment, we formulate robot and object motions as unified 4D point trajectories. To overcome retargeting complexity, our Keypoints Tracker tracks only sparse critical points (base, hands, and object), entirely bypassing the error-amplifying retargeting process. To maintain natural gaits despite these sparse signals, we utilize the latent space of a Behavior Foundation Model (BFM) as the tracker's search domain. Using a progressive training strategy, Imagine2Real learns robust behaviors with simple tracking rewards, enabling zero-shot physical deployment within a motion capture(mocap) system.
Jiahe Chen, ZiRui Wang, Feiyu Jia +7
Zhejiang University · Shanghai AI Laboratory · The Chinese University of Hong Kong
Physics-based human-object interaction has achieved robust single-agent manipulation skills, yet extending them to multi-agent cooperative tasks remains challenging. Existing approaches typically adapt interaction policies through task-specific fine-tuning, which entangles low-level contact-rich execution with high-level coordination and limits reuse across object geometries, interaction types, and team sizes. We propose a hierarchical framework that converts a single-agent HOI policy into a reusable Object-oriented Motion Skill. Specifically, we reinterpret teacher rollouts as object-oriented action supervision by extracting short-horizon object-proxy motions from executed trajectories, and distill task-specific teachers into a low-level skill operating in an Object-oriented Action Space. For downstream tasks, the distilled skill is frozen as a reusable executor, while a high-level policy coordinates multiple agents by generating region-wise object-oriented actions conditioned on the shared object, task goal, agent states, and local manipulation regions. This formulation shifts multi-agent HOI learning from direct contact-rich full-body control to compact object-level proxy-motion coordination. Experiments on diverse HOI tasks show that the distilled Object-oriented Motion Skill supports robust proxy-motion execution and enables composable policy learning across different interaction types, object geometries, and team sizes.