World2HOSI: A Multi-Expert Harness for Long-Horizon Human-Object-Scene Interaction Generation

Jiushu Li; Haoran Wang; Yumeng Liu; Zhiyang Dou; Yuan Liu; Yuexin Ma; Cheng Lin

cs.CVOct 20, 2026

World2HOSI: A Multi-Expert Harness for Long-Horizon Human-Object-Scene Interaction Generation

Authors: Jiushu Li, Haoran Wang, Yumeng Liu, Zhiyang Dou, Yuan Liu, Yuexin Ma, Cheng Lin

Organizations: Macau University of Science and Technology · University of Science and Technology of China · Massachusetts Institute of Technology · The Hong Kong University of Science and Technology ·

Abstract

Long-horizon human-object-scene interaction (HOSI) is central to embodied AI, animation, and game development. Existing human-object interaction (HOI) and human-scene interaction (HSI) models excel in their respective domains, but none alone covers tasks requiring multiple interaction skills. Training unified HOSI models is hindered by scarce and costly data. We introduce World2HOSI, a multi-expert orchestration harness that generates continuous HOSI data from 3D Gaussian Splatting (3DGS) scenes and natural language instructions. Our key insight is that long-horizon HOSI hinges not on a stronger single model but on orchestration. Given a shared, structured scene state and an orchestration harness grounded in it, existing experts can jointly accomplish tasks beyond the reach of any of them alone. World2HOSI realizes this insight as a closed loop around a persistent scene state built from decoupled background and object assets. A harness plans action stages over this state, invokes experts via their native conditioning interfaces, and writes each action's effects back, so that every invocation is grounded in the evolving world. To reduce physical artifacts, bounded residual optimization and contact-driven physics simulation further ensure the physical plausibility of human motion and object manipulation, respectively, while preserving interaction intent. Experiments and a perceptual user study show that World2HOSI achieves the best task completion and interaction quality on the evaluated HOSI tasks, while generating more diverse data.

Figures & tables

Figure 1: Generating diverse long-horizon HOSI in reconstructed 3D worlds. World2HOSI orchestrates complementary motion experts from natural-language instructions to generate continuous sequences of navigation, object manipulation, and scene interaction in a persistently updated world.

Figure 2: World2HOSI closes the loop around a persistent scene state. (a) The ISP organizes background and object assets. (b) Task Planner grounds each action stage in the current state. (c) Experts generate motion, whose outcomes update the ISP before the next invocation. (d) Optional physics-based refinement supports contact-driven execution. State write-back keeps subsequent targets consistent with earlier actions.

Method	Task Completion			Interaction Plausibility				Human Motion Quality
Method	$T_{xy}\downarrow$	$S_{t}\uparrow$	$S_{g}\uparrow$	$\mathrm{OCP}\uparrow$	$P_{f}\downarrow$	$\mathrm{NCOD}\downarrow$	$P_{\mathrm{os}}\downarrow$	$P_{\mathrm{ho}}\downarrow$	$\mathrm{FS}\downarrow$	$O_{\mathrm{feet}}\downarrow$
InfBaGel	36.02	42.54	44.61	8.52	47.99	146.91	2.201	2.433	0.048	4.77
HOSIG	44.53	65.33	48.16	78.14	85.44	2.97	2.077	2.069	0.823	5.27
HOI-FHLI	23.14	62.54	64.68	77.29	43.36	2.88	1.105	1.554	0.467	4.61
Ours	17.88	76.93	75.61	84.94	24.92	2.08	0.708	1.309	0.255	4.27

Table 1: Quantitative results on the shared HOSI tasks. Compared with the external baselines, our method achieves superior task completion and interaction quality.

Figure 3: Qualitative comparison with external methods. We compare two object-carrying tasks, together with a more complex multi-stage task.

Metric $\downarrow$	Concat. (A0)	Bi-Blend (A1)	Proj. (A2)
Joint Disp. (cm)	1.135	0.840	0.593
Joint Angle ( ∘ )	4.615	1.224	0.397
Velocity Jump (cm/s)	33.437	0.637	0.208

Table 3: Ablation of cross-expert reconciliation. We compare different boundary treatments to assess the improvement in motion smoothness from cross-expert reconciliation.

	$E_{h}$	$E_{o}$
Ours	5.39	3.07

Table 4: Human-object errors between kinematic reference and physics-based motion.

Figure 4: Qualitative comparison between kinematic reference motion and physics-based motion. The residual correction policy effectively removes artifacts such as floating and interpenetration, while also improving grasping poses.

Figure 5: Long-horizon interaction with persistent scene state. Top: relocating a chair, drinking coffee, and returning to sit. Bottom: moving books, repositioning a stool, and sitting.

Figure 6: Comparison with the baselines in the perceptual user study.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Long-horizon interaction tasks with state-update dependencies. Top: Removing an obstacle blocking a doorway before entering the room. Bottom: Moving a chair to a new location before sitting on it. Both examples illustrate how World2HOSI uses scene states updated by preceding interactions to provide consistent spatial conditions and interaction constraints for subsequent motion generation.

Evidence	Scoring rule
Human-target distance	Full credit when the distance is at most 10 cm, zero at or beyond 60 cm, and linear interpolation between these thresholds.
Approach / arrival distance	A target is considered approached within 50 cm and reached within 30 cm.
Initiation / full-credit displacement	Manipulation is considered initiated when the object has moved 1 cm; the initiation component receives full credit at 3 cm.
Motion speed / stopping observation	The object is considered in motion at speeds of at least 0.02 m/s. Placement or stopping is recognized when its position varies by no more than 5 cm over a continuous 0.2 s interval.
Object target region	Full credit when the final object position is within 0.50 m of the target, and zero when it is more than 1.00 m away.
Sitting pose change	The full-credit scales are a 0.15 m decrease in pelvis height and $60^{\circ}$ of knee flexion.

Appendix

Table 5: Evidence and scoring rules for the task progress score

S_{t}

HOI Generation

Human-object interaction generation creates 3D motion, grasps, and videos of people manipulating or coordinating with objects.

See all 84 papers

InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinement
Apr 6, 2026Yude Zou, Junji Gong, Xing Gao +3HOI Generation
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Sichuan University +2
OneHOI: Unifying Human-Object Interaction Generation and Editing
Apr 15, 2026Jiun Tian Hoe, Weipeng Hu, Xudong Jiang +2HOI Generation Diffusion Transformer
Nanyang Technological University · Sun Yat-sen University · VinUniversity +1
Learning to Generate Human-Human-Object Interactions from Textual Descriptions
Nov 25, 2025Jeonghyeon Na, Sangwon Baik, Inhee Lee +2HOI Generation Human Motion Generation
Seoul National University

Apr 6, 2026J/K move · Enter open · S save

InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinement

Yude Zou, Junji Gong, Xing Gao, Zixuan Li+2

Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Sichuan University+2

HOI Generation

Open paper AskPDF arXiv

Human-object-scene interactions (HOSI) generation has broad applications in embodied AI, simulation, and animation. Unlike human-object interaction (HOI) and human-scene interaction (HSI), HOSI generation requires reasoning over dynamic object-scene changes, yet suffers from limited annotated data. To address these issues, we propose a coarse-to-fine instruction-conditioned interaction generation framework that is explicitly aligned with the iterative denoising process of a consistency model. In particular, we adopt a dynamic perception strategy that leverages trajectories from the preceding refinement to update scene context and condition subsequent refinement at each denoising step of consistency model, yielding consistent interactions. To further reduce physical artifacts, we introduce a bump-aware guidance that mitigates collisions and penetrations during sampling without requiring fine-grained scene geometry, enabling real-time generation. To overcome data scarcity, we design a hybrid training startegy that synthesizes pseudo-HOSI samples by injecting voxelized scene occupancy into HOI datasets and jointly trains with high-fidelity HSI data, allowing interaction learning while preserving realistic scene awareness. Extensive experiments demonstrate that our method achieves state-of-the-art performance in both HOSI and HOI generation, and strong generalization to unseen scenes. Project page: https://yudezou.github.io/InfBaGel-page/

Abstract

HOI Generation

Explore similar work

InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinement

OneHOI: Unifying Human-Object Interaction Generation and Editing

Learning to Generate Human-Human-Object Interactions from Textual Descriptions