cs.ROSep 6, 2026
SaveUnifying Physics-Based Humanoid Interaction with a Context-Conditioned Interaction Prior
Organizations: The Chinese University of Hong Kong, Department of Computer Science and Engineering, Hong Kong, China · The Chinese University of Hong Kong, Department of Information Engineering, Hong Kong, China · Monash University, Department of Data Science and AI, Clayton, Australia
Abstract
Developing unified physics-based humanoid controllers that can navigate complex 3D scenes and manipulate objects remains a longstanding challenge. Existing approaches are often specialized for either locomotion or object-centric manipulation, or rely on task-specific reward engineering that does not scale well across diverse behaviors. We present CHIP, a unified, physics-grounded framework for learning reusable humanoid interaction skills from heterogeneous motion data. Central to our approach is a conditional interaction prior that models a context-dependent distribution over these skills within a shared discrete space. Our method is trained in three stages. We first learn physics-based motion-imitation policies that acquire grounded teacher behaviors from heterogeneous interaction data. We then distill these behaviors into a context-conditioned interaction prior that captures reusable motion structure across locomotion and manipulation. Finally, we initialize downstream task policies from the pretrained prior and adapt them through prior-regularized online RL post-training. Experiments on a diverse suite of humanoid interaction tasks show that our approach supports scene-aware locomotion, contact-rich object manipulation, and compositional behaviors such as environment-aware object transport and long-horizon skill sequencing, while producing smooth transitions and physically plausible motion.
Figures & tables
Figure 1. CHIP enables multi-stage humanoid interaction in complex 3D environments. Representative rollouts demonstrate whole-body collision avoidance when passing sideways through narrow gaps, contact-rich object manipulation, and compositional capabilities such as obstacle-aware object carrying within a single continuous behavior. Teaser figure showing qualitative examples of multi-stage humanoid interaction behaviors, where the humanoid composes navigation and manipulation skills across multiple stages of a task.
Figure 2. Overview of the proposed framework. Starting from heterogeneous HSI and HOI motion data, Stage I trains physics-based tracking policies to reproduce scene- and object-interaction behaviors. Stage II distills their behaviors into a shared discrete skill representation and a context-conditioned interaction prior using multimodal scene, object, and task inputs. Stage III initializes a downstream task policy from the pretrained prior and adapts the policy with reinforcement learning, while KL regularization toward the frozen prior preserves the learned interaction structure and motion quality. Left-to-right overview of the three-stage framework. Heterogeneous scene- and object-interaction motions and their conditioning signals feed physics-based imitation experts. Their behaviors are distilled into a context-conditioned generative skill prior, which initializes an RL task policy regularized toward the frozen prior for navigation, manipulation, and composite tasks.
Figure 3. Architecture of the generative interaction prior. Teacher behaviors are first encoded into discrete skill tokens, while multimodal interaction contexts and task goals are tokenized through a unified conditioning interface. A transformer-based prior predicts context-conditioned skill tokens, which are decoded into low-level humanoid actions. Architecture diagram in which proprioceptive-history tokens and randomly masked conditioning tokens enter a transformer categorical prior that predicts a discrete skill token. During training, an encoder quantizes teacher motion features through a codebook; the selected codebook embedding and current proprioception are passed to a decoder that outputs humanoid actions.
Figure 4. Qualitative results on scene navigation. Representative snapshots show coordinated whole-body motion in cluttered scenes, including torso rotation and sideways stepping to pass through narrow gaps while maintaining stable, goal-directed locomotion. Two four-frame navigation sequences viewed from different angles. In each sequence, the humanoid rotates its torso and steps sideways to pass through a narrow gap between pieces of furniture without colliding with them.
Figure 5. Qualitative results on compositional capabilities. Representative snapshots show compositional behaviors acquired through RL post-training: the humanoid transports an object while remaining aware of the surrounding environment, bypassing obstacles by adapting its whole-body motion. Two four-frame sequences of a humanoid transporting a large crate through cluttered rooms toward a green goal marker. Red height-scan points indicate nearby scene geometry as the humanoid changes its body orientation and carrying posture to move around obstacles.
| Scene Nav. | Object Reloc. | Compositional | |||||||||
| Method | Succ. | Dist. | Coll. | Fail. | Succ. | Dist. | Fail. | Succ. | Dist. | Coll. | Fail. |
| TokenHSI ( Pan et al., 2025 ) | 93.3 | 0.05 | 0.03 | 1.6 | 84.8 | 0.14 | 1.2 | 66.8 | 0.19 | 0.04 | 1.8 |
| MaskedMimic ( Tessler et al., 2024 ) | 33.8 | 0.36 | 0.13 | 5.1 | 44.1 | 0.36 | 8.4 | 0.0 | 0.52 | 0.15 | 14.1 |
| Ours | 90.1 | 0.05 | 0.02 | 1.5 | 88.2 | 0.10 | 1.0 | 71.4 | 0.13 | 0.02 | 1.6 |
Table 1. Quantitative results on heterogeneous and compositional humanoid interaction tasks. We report success and failure rates (%), final distance-to-goal (m), and collision-frame fractions for Scene Navigation, Object Relocation, and the Compositional Task.
Figure 6. Diverse object interactions. Qualitative results across the evaluated object instances and interaction types demonstrate coherent whole-body control. Four examples of whole-body object interaction on a blue grid floor. From left to right, the humanoid interacts with a chair, bends over a table, handles a large bin, and carries a rectangular crate.
Figure 7. Ablation on scene-context conditioning. Without scene-context conditioning, RL can exploit the task rewards and learn irregular behaviors, such as moving backward, instead of maintaining a natural walking gait in cluttered environments. Side-by-side multi-exposure trajectories in the same furnished room. With the scene-context prior, the humanoid walks naturally around furniture toward a green goal; without it, the humanoid reaches the goal using an irregular backward-facing trajectory.
| Setting | Nav. Succ. | Nav. Fail. | Comp. Succ. | Comp. Fail. |
| In-distribution | 91.3 | 1.8 | 69.8 | 4.8 |
| Held-out scenes | 86.1 | 5.5 | 60.2 | 10.5 |
Table 2. Held-out scene generalization of CHIP. We compare randomized in-distribution evaluation with ten scenes excluded from all training stages, while keeping the object-instance pool and task and goal distributions unchanged.
| Variant | RL Post-train. | Prior Reg. Loss | Inter. Context | Scene Nav. | Object Reloc. | Comp. Task |
| Pretrained prior only | ✗ | ✗ | ✓ | 48.9 | 55.2 | 4.8 |
| w/o prior reg. | ✓ | ✗ | ✗ | 94.2 | 0.0 | 0.0 |
| w/o context prior | ✓ | ✓ | ✗ | 92.1 | 0.0 | 0.0 |
| Full | ✓ | ✓ | ✓ | 90.1 | 88.2 | 71.4 |
Table 3. Component-wise ablation of CHIP. We report success rates on downstream tasks to evaluate the contributions of RL post-training, prior regularization, and context-conditioned interaction prior modeling.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Data source | Files | Weight | Avg. Len. (s) | Has Object | Has Scene | Total Len. (h) |
| locomotion | 6 | 0.05 | 389.06 | No | No | 0.65 |
| box transport | 120 | 0.05 | 22.00 | Yes | No | 0.73 |
| TRUMANS (idle) | 1486 | 0.45 | 21.11 | No | Yes | 8.71 |
| OMOMO (subset) | 1214 | 0.45 | 6.14 | Yes | No | 2.07 |
| Total | 2826 | 1.00 | 15.50 | – | – | 12.17 |
Table 4. Motion data statistics. We report the number of files, training sampling weight, sequence duration, and scene/object availability for each data source after applying the training-time motion filters.
| Hyperparameter | Value |
| RL algorithm | PPO |
| Optimizer | AdamW |
| Tracking actor | MLP , ELU, tanh output |
| Tracking critic | MLP , ELU, linear value output |
| Learning rate | |
| Learning-rate schedule | Adaptive |
Table 5. Key hyperparameters for Stage I motion imitation.
| Hyperparameter | Value |
| VQ-VAE encoder | MLP ; ReLU |
| VQ-VAE decoder | MLP ; ReLU |
| Codebook | ; 64-D embeddings; one quantizer |
| Commitment weight | |
| EMA decay | |
| Prior backbone | 4-layer Transformer; model dimension ; 8 attention heads |
Table 6. Stage II skill-distillation and prior-learning hyperparameters.
| Hyperparameter | Value |
| RL algorithm | PPO |
| Actor backbone | 4-layer Transformer; model dimension ; 8 attention heads |
| Actor FFN / output | FFN dimension ; GELU; dropout ; 1024-way categorical skill distribution |
| Actor / prior history input | 5 proprioceptive frames |
| Policy initialization | Pretrained interaction prior |
| Trainable modules | Task policy |
Table 7. Stage III RL post-training hyperparameters.
| Reward component | Weight | |
| Anchor position | ||
| Anchor orientation | ||
| Relative body position | ||
| Relative body orientation | ||
| Global body linear velocity | ||
| Global body angular velocity |
Table 8. Stage I reward weights and fixed RBF bandwidths.
| Task | Observation components | Reward weights |
| Scene Navigation | Local heightmap covering ; planar target displacement in the humanoid heading frame, stored in a shared padded 10-D target tensor. | , , , , , , ; common penalties: , , , . |
| Object Relocation | Local heightmap; heading-frame object goal; 96-D body–object interaction geometry; 512-D object BPS; optional normalized 512-D CLIP text feature. | , , , ; common penalties: , , , . |
Table 9. Observation components and reward weights for downstream tasks. We list task-specific observations and the reward weights used during RL post-training. Common penalties are shared across tasks.