Organizations: Macau University of Science and Technology · University of Science and Technology of China · Massachusetts Institute of Technology · The Hong Kong University of Science and Technology · ShanghaiTech University
Long-horizon human-object-scene interaction (HOSI) is central to embodied AI, animation, and game development. Existing human-object interaction (HOI) and human-scene interaction (HSI) models excel in their respective domains, but none alone covers tasks requiring multiple interaction skills. Training unified HOSI models is hindered by scarce and costly data. We introduce World2HOSI, a multi-expert orchestration harness that generates continuous HOSI data from 3D Gaussian Splatting (3DGS) scenes and natural language instructions. Our key insight is that long-horizon HOSI hinges not on a stronger single model but on orchestration. Given a shared, structured scene state and an orchestration harness grounded in it, existing experts can jointly accomplish tasks beyond the reach of any of them alone. World2HOSI realizes this insight as a closed loop around a persistent scene state built from decoupled background and object assets. A harness plans action stages over this state, invokes experts via their native conditioning interfaces, and writes each action's effects back, so that every invocation is grounded in the evolving world. To reduce physical artifacts, bounded residual optimization and contact-driven physics simulation further ensure the physical plausibility of human motion and object manipulation, respectively, while preserving interaction intent. Experiments and a perceptual user study show that World2HOSI achieves the best task completion and interaction quality on the evaluated HOSI tasks, while generating more diverse data.
Figures & tables
Figure 1: Generating diverse long-horizon HOSI in reconstructed 3D worlds. World2HOSI orchestrates complementary motion experts from natural-language instructions to generate continuous sequences of navigation, object manipulation, and scene interaction in a persistently updated world.
Figure 2: World2HOSI closes the loop around a persistent scene state. (a) The ISP organizes background and object assets. (b) Task Planner grounds each action stage in the current state. (c) Experts generate motion, whose outcomes update the ISP before the next invocation. (d) Optional physics-based refinement supports contact-driven execution. State write-back keeps subsequent targets consistent with earlier actions.
Method
Task Completion
Interaction Plausibility
Human Motion Quality
Txy↓
St↑
Sg↑
OCP↑
Pf↓
NCOD↓
Pos↓
Pho↓
FS↓
Ofeet↓
InfBaGel
36.02
42.54
44.61
8.52
47.99
146.91
2.201
2.433
0.048
4.77
HOSIG
44.53
65.33
48.16
78.14
85.44
2.97
2.077
2.069
0.823
5.27
HOI-FHLI
23.14
62.54
64.68
77.29
43.36
2.88
1.105
1.554
0.467
4.61
Ours
17.88
76.93
75.61
84.94
24.92
2.08
0.708
1.309
0.255
4.27
Table 1: Quantitative results on the shared HOSI tasks. Compared with the external baselines, our method achieves superior task completion and interaction quality.
Figure 3: Qualitative comparison with external methods. We compare two object-carrying tasks, together with a more complex multi-stage task.
Metric ↓
Concat. (A0)
Bi-Blend (A1)
Proj. (A2)
Joint Disp. (cm)
1.135
0.840
0.593
Joint Angle ( ∘ )
4.615
1.224
0.397
Velocity Jump (cm/s)
33.437
0.637
0.208
Table 3: Ablation of cross-expert reconciliation. We compare different boundary treatments to assess the improvement in motion smoothness from cross-expert reconciliation.
Eh
Eo
Ours
5.39
3.07
Table 4: Human-object errors between kinematic reference and physics-based motion.
Figure 4: Qualitative comparison between kinematic reference motion and physics-based motion. The residual correction policy effectively removes artifacts such as floating and interpenetration, while also improving grasping poses.
Figure 5: Long-horizon interaction with persistent scene state. Top: relocating a chair, drinking coffee, and returning to sit. Bottom: moving books, repositioning a stool, and sitting.
Figure 6: Comparison with the baselines in the perceptual user study.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Long-horizon interaction tasks with state-update dependencies. Top: Removing an obstacle blocking a doorway before entering the room. Bottom: Moving a chair to a new location before sitting on it. Both examples illustrate how World2HOSI uses scene states updated by preceding interactions to provide consistent spatial conditions and interaction constraints for subsequent motion generation.
Evidence
Scoring rule
Human-target distance
Full credit when the distance is at most 10 cm, zero at or beyond 60 cm, and linear interpolation between these thresholds.
Approach / arrival distance
A target is considered approached within 50 cm and reached within 30 cm.
Initiation / full-credit displacement
Manipulation is considered initiated when the object has moved 1 cm; the initiation component receives full credit at 3 cm.
Motion speed / stopping observation
The object is considered in motion at speeds of at least 0.02 m/s. Placement or stopping is recognized when its position varies by no more than 5 cm over a continuous 0.2 s interval.
Object target region
Full credit when the final object position is within 0.50 m of the target, and zero when it is more than 1.00 m away.
Sitting pose change
The full-credit scales are a 0.15 m decrease in pelvis height and 60∘ of knee flexion.
Appendix
Table 5: Evidence and scoring rules for the task progress score St .
Human-object-scene interactions (HOSI) generation has broad applications in embodied AI, simulation, and animation. Unlike human-object interaction (HOI) and human-scene interaction (HSI), HOSI generation requires reasoning over dynamic object-scene changes, yet suffers from limited annotated data. To address these issues, we propose a coarse-to-fine instruction-conditioned interaction generation framework that is explicitly aligned with the iterative denoising process of a consistency model. In particular, we adopt a dynamic perception strategy that leverages trajectories from the preceding refinement to update scene context and condition subsequent refinement at each denoising step of consistency model, yielding consistent interactions. To further reduce physical artifacts, we introduce a bump-aware guidance that mitigates collisions and penetrations during sampling without requiring fine-grained scene geometry, enabling real-time generation. To overcome data scarcity, we design a hybrid training startegy that synthesizes pseudo-HOSI samples by injecting voxelized scene occupancy into HOI datasets and jointly trains with high-fidelity HSI data, allowing interaction learning while preserving realistic scene awareness. Extensive experiments demonstrate that our method achieves state-of-the-art performance in both HOSI and HOI generation, and strong generalization to unseen scenes. Project page: https://yudezou.github.io/InfBaGel-page/
Yude Zou, Junji Gong, Xing Gao +3
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Sichuan University +2
Human-Object Interaction (HOI) modelling captures how humans act upon and relate to objects, typically expressed as <person, action, object> triplets. Existing approaches split into two disjoint families: HOI generation synthesises scenes from structured triplets and layout, but fails to integrate mixed conditions like HOI and object-only entities; and HOI editing modifies interactions via text, yet struggles to decouple pose from physical contact and scale to multiple interactions. We introduce OneHOI, a unified diffusion transformer framework that consolidates HOI generation and editing into a single conditional denoising process driven by shared structured interaction representations. At its core, the Relational Diffusion Transformer (R-DiT) models verb-mediated relations through role- and instance-aware HOI tokens, layout-based spatial Action Grounding, a Structured HOI Attention to enforce interaction topology, and HOI RoPE to disentangle multi-HOI scenes. Trained jointly with modality dropout on our HOI-Edit-44K, along with HOI and object-centric datasets, OneHOI supports layout-guided, layout-free, arbitrary-mask, and mixed-condition control, achieving state-of-the-art results across both HOI generation and editing. Code is available at https://jiuntian.github.io/OneHOI/.
Jiun Tian Hoe, Weipeng Hu, Xudong Jiang +2
Nanyang Technological University · Sun Yat-sen University · VinUniversity +1
The way humans interact with each other, including interpersonal distances, spatial configuration, and motion, varies significantly across different situations. To enable machines to understand such complex, context-dependent behaviors, it is essential to model multiple people in relation to the surrounding scene context. In this paper, we present a novel research problem to model the correlations between two people engaged in a shared interaction involving an object. We refer to this formulation as Human-Human-Object Interactions (HHOIs). To overcome the lack of dedicated datasets for HHOIs, we present a newly captured HHOIs dataset and a method to synthesize HHOI data by leveraging image generative models. As an intermediary, we obtain individual human-object interaction (HOIs) and human-human interaction (HHIs) from the HHOIs, and with these data, we train an text-to-HOI and text-to-HHI model using score-based diffusion model. Finally, we present a unified generative framework that integrates the two individual model, capable of synthesizing complete HHOIs in a single advanced sampling process. Our method extends HHOI generation to multi-human settings, enabling interactions involving more than two individuals. Experimental results show that our method generates realistic HHOIs conditioned on textual descriptions, outperforming previous approaches that focus only on single-human HOIs. Furthermore, we introduce multi-human motion generation involving objects as an application of our framework.