Unifying Physics-Based Humanoid Interaction with a Context-Conditioned Interaction Prior
Authors: Jianan Li, Xiao Chen, Tien-Tsin Wong
Organizations: The Chinese University of Hong Kong, Department of Computer Science and Engineering, Hong Kong, China · The Chinese University of Hong Kong, Department of Information Engineering, Hong Kong, China · Monash University, Department of Data Science and AI, Clayton, Australia
Developing unified physics-based humanoid controllers that can navigate complex 3D scenes and manipulate objects remains a longstanding challenge. Existing approaches are often specialized for either locomotion or object-centric manipulation, or rely on task-specific reward engineering that does not scale well across diverse behaviors. We present CHIP, a unified, physics-grounded framework for learning reusable humanoid interaction skills from heterogeneous motion data. Central to our approach is a conditional interaction prior that models a context-dependent distribution over these skills within a shared discrete space. Our method is trained in three stages. We first learn physics-based motion-imitation policies that acquire grounded teacher behaviors from heterogeneous interaction data. We then distill these behaviors into a context-conditioned interaction prior that captures reusable motion structure across locomotion and manipulation. Finally, we initialize downstream task policies from the pretrained prior and adapt them through prior-regularized online RL post-training. Experiments on a diverse suite of humanoid interaction tasks show that our approach supports scene-aware locomotion, contact-rich object manipulation, and compositional behaviors such as environment-aware object transport and long-horizon skill sequencing, while producing smooth transitions and physically plausible motion.
Figures & tables
Figure 1. CHIP enables multi-stage humanoid interaction in complex 3D environments. Representative rollouts demonstrate whole-body collision avoidance when passing sideways through narrow gaps, contact-rich object manipulation, and compositional capabilities such as obstacle-aware object carrying within a single continuous behavior. Teaser figure showing qualitative examples of multi-stage humanoid interaction behaviors, where the humanoid composes navigation and manipulation skills across multiple stages of a task.
Figure 2. Overview of the proposed framework. Starting from heterogeneous HSI and HOI motion data, Stage I trains physics-based tracking policies to reproduce scene- and object-interaction behaviors. Stage II distills their behaviors into a shared discrete skill representation and a context-conditioned interaction prior using multimodal scene, object, and task inputs. Stage III initializes a downstream task policy from the pretrained prior and adapts the policy with reinforcement learning, while KL regularization toward the frozen prior preserves the learned interaction structure and motion quality. Left-to-right overview of the three-stage framework. Heterogeneous scene- and object-interaction motions and their conditioning signals feed physics-based imitation experts. Their behaviors are distilled into a context-conditioned generative skill prior, which initializes an RL task policy regularized toward the frozen prior for navigation, manipulation, and composite tasks.
Figure 3. Architecture of the generative interaction prior. Teacher behaviors are first encoded into discrete skill tokens, while multimodal interaction contexts and task goals are tokenized through a unified conditioning interface. A transformer-based prior predicts context-conditioned skill tokens, which are decoded into low-level humanoid actions. Architecture diagram in which proprioceptive-history tokens and randomly masked conditioning tokens enter a transformer categorical prior that predicts a discrete skill token. During training, an encoder quantizes teacher motion features through a codebook; the selected codebook embedding and current proprioception are passed to a decoder that outputs humanoid actions.
Figure 4. Qualitative results on scene navigation. Representative snapshots show coordinated whole-body motion in cluttered scenes, including torso rotation and sideways stepping to pass through narrow gaps while maintaining stable, goal-directed locomotion. Two four-frame navigation sequences viewed from different angles. In each sequence, the humanoid rotates its torso and steps sideways to pass through a narrow gap between pieces of furniture without colliding with them.
Figure 5. Qualitative results on compositional capabilities. Representative snapshots show compositional behaviors acquired through RL post-training: the humanoid transports an object while remaining aware of the surrounding environment, bypassing obstacles by adapting its whole-body motion. Two four-frame sequences of a humanoid transporting a large crate through cluttered rooms toward a green goal marker. Red height-scan points indicate nearby scene geometry as the humanoid changes its body orientation and carrying posture to move around obstacles.
Scene Nav.
Object Reloc.
Compositional
Method
Succ. ↑
Dist. ↓
Coll. ↓
Fail. ↓
Succ. ↑
Dist. ↓
Fail. ↓
Succ. ↑
Dist. ↓
Coll. ↓
Fail. ↓
TokenHSI ( Pan et al., 2025 )
93.3
0.05
0.03
1.6
84.8
0.14
1.2
66.8
0.19
0.04
1.8
MaskedMimic ( Tessler et al., 2024 )
33.8
0.36
0.13
5.1
44.1
0.36
8.4
0.0
0.52
0.15
14.1
Ours
90.1
0.05
0.02
1.5
88.2
0.10
1.0
71.4
0.13
0.02
1.6
Table 1. Quantitative results on heterogeneous and compositional humanoid interaction tasks. We report success and failure rates (%), final distance-to-goal (m), and collision-frame fractions for Scene Navigation, Object Relocation, and the Compositional Task.
Figure 6. Diverse object interactions. Qualitative results across the evaluated object instances and interaction types demonstrate coherent whole-body control. Four examples of whole-body object interaction on a blue grid floor. From left to right, the humanoid interacts with a chair, bends over a table, handles a large bin, and carries a rectangular crate.
Figure 7. Ablation on scene-context conditioning. Without scene-context conditioning, RL can exploit the task rewards and learn irregular behaviors, such as moving backward, instead of maintaining a natural walking gait in cluttered environments. Side-by-side multi-exposure trajectories in the same furnished room. With the scene-context prior, the humanoid walks naturally around furniture toward a green goal; without it, the humanoid reaches the goal using an irregular backward-facing trajectory.
Setting
Nav. Succ. ↑
Nav. Fail. ↓
Comp. Succ. ↑
Comp. Fail. ↓
In-distribution
91.3
1.8
69.8
4.8
Held-out scenes
86.1
5.5
60.2
10.5
Table 2. Held-out scene generalization of CHIP. We compare randomized in-distribution evaluation with ten scenes excluded from all training stages, while keeping the object-instance pool and task and goal distributions unchanged.
Variant
RL Post-train.
Prior Reg. Loss
Inter. Context
Scene Nav.
Object Reloc.
Comp. Task
Pretrained prior only
✗
✗
✓
48.9
55.2
4.8
w/o prior reg.
✓
✗
✗
94.2
0.0
0.0
w/o context prior
✓
✓
✗
92.1
0.0
0.0
Full
✓
✓
✓
90.1
88.2
71.4
Table 3. Component-wise ablation of CHIP. We report success rates on downstream tasks to evaluate the contributions of RL post-training, prior regularization, and context-conditioned interaction prior modeling.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Data source
Files
Weight
Avg. Len. (s)
Has Object
Has Scene
Total Len. (h)
locomotion
6
0.05
389.06
No
No
0.65
box transport
120
0.05
22.00
Yes
No
0.73
TRUMANS (idle)
1486
0.45
21.11
No
Yes
8.71
OMOMO (subset)
1214
0.45
6.14
Yes
No
2.07
Total
2826
1.00
15.50
–
–
12.17
Appendix
Table 4. Motion data statistics. We report the number of files, training sampling weight, sequence duration, and scene/object availability for each data source after applying the training-time motion filters.
Hyperparameter
Value
RL algorithm
PPO
Optimizer
AdamW
Tracking actor
MLP [512,256,128] , ELU, tanh output
Tracking critic
MLP [512,256,128] , ELU, linear value output
Learning rate
1×10−3
Learning-rate schedule
Adaptive
Appendix
Table 5. Key hyperparameters for Stage I motion imitation.
Hyperparameter
Value
VQ-VAE encoder
MLP [1024,1024,768,512] ; ReLU
VQ-VAE decoder
MLP [1024,1024,768,512] ; ReLU
Codebook
K=1024 ; 64-D embeddings; one quantizer
Commitment weight βcom
0.25
EMA decay
0.99
Prior backbone
4-layer Transformer; model dimension 512 ; 8 attention heads
Appendix
Table 6. Stage II skill-distillation and prior-learning hyperparameters.
Hyperparameter
Value
RL algorithm
PPO
Actor backbone
4-layer Transformer; model dimension 512 ; 8 attention heads
Table 9. Observation components and reward weights for downstream tasks. We list task-specific observations and the reward weights used during RL post-training. Common penalties are shared across tasks.
Achieving human-level dexterity in complex, unstructured environments requires the seamless integration of whole-body scene interaction and dexterous object manipulation skills. While existing physics-based controllers generate physically plausible behaviors in each domain, they largely address these two capabilities independently. In this paper, we present Co2Skill that integrates scene interaction and dexterous manipulation through a unified policy formulation. Built on a pretrained motion prior, the policy uses task and phase dependent observation masks to select information relevant to the current interaction goals. We introduce a goal-conditioned loco-manipulation curriculum that combines partial reference guidance for precision with exploration from varied initial states while allowing goal-directed execution beyond the demonstrated trajectories. We further introduce a cross-task curriculum that jointly trains individual skills and selected task sequences, preserving physical states across task boundaries and maintaining grasps during subsequent scene interactions. Together, these support sequential task execution and simultaneous scene interaction with object manipulation. We evaluate sitting, standing, climbing, stair traversal, and goal-directed manipulation, together with sequential execution and with random different conditions. Additionally, we demonstrate skill compositions in indoor environments, illustrating their integration within the same control formulation.
Jeonghwan Kim, Hyeonwoo Kim, Hanbyul Joo
Department of Computer Science Seoul National University
Humanoid robots operating in unstructured environments must combine robust whole-body control with the ability to perceive and physically interact with surrounding objects. While large-scale human motion data provides powerful priors for natural and versatile humanoid control, effectively transferring such priors to perception-driven object interaction remains challenging. To address this bottleneck, we propose a framework that extends the recently proposed Generative Pretrained Controller (GPC) from general human motion to full-body humanoid-environment interaction. First, we adapt GPC into interaction experts conditioned on scene affordance cues and privileged state information. This adaptation begins with interaction and affordance pre-training on human motions paired with support scenes. These experts leverage the pretrained human motion prior while learning task-specific contact behaviors, including reaching toward objects, grasping environmental supports for stabilization, and pushing movable objects. Second, we introduce a perception-driven student that retains the pretrained GPC policy and distills interaction skills from the experts using onboard sensory observations. To bridge the gap between privileged expert observations and sensory inputs, we train the student's perception not only to reconstruct expert observations, but also to preserve the expert's motion decisions and resulting joint targets. Notably, our experiments across multiple whole-body interaction tasks demonstrate that large-scale generative human motion priors provide an effective foundation for learning deployable policies for humanoid interactions in contact-rich real-world environments.
Anujith Muraleedharan, Abdul Ahad Butt, Nolan Fey +5
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates. · Massachusetts Institute of Technology (MIT), Cambridge, MA, USA. · Minerva Humanoids, Inc.
Interaction is one of the core abilities of humanoid robots. However, most existing frameworks focus on non-interactive whole-body control, which limits their practical applicability. In this work, we develop InterReal, a unified physics-based imitation learning framework for Real-world human-object Interaction (HOI) control. InterReal enables humanoid robots to track HOI reference motions, facilitating the learning of fine-grained interactive skills and their deployment in real-world settings. Within this framework, we first introduce a HOI motion data augmentation scheme with hand-object contact constraints, and utilize the augmented motions to improve policy stability under object perturbations. Second, we propose an automatic reward learner to address the challenge of large-scale reward shaping. A meta-policy guided by critical tracking error metrics explores and allocates reward signals to the low-level reinforcement learning objective, which enables more effective learning of interactive policies. Experiments on HOI tasks of box-picking and box-pushing demonstrate that InterReal achieves the best tracking accuracy and the highest task success rate compared to recent baselines. Furthermore, we validate the framework on the real-world robot Unitree G1, which demonstrates its practical effectiveness and robustness beyond simulation.
Dayang Liang, Yuhang Lin, Xinzhe Liu +3
Xiamen University · Zhejiang University · ShanghaiTech University +1