cs.RONov 18, 2026

Inhabitant: An Agent Framework for Long-Horizon Human-Scene Interaction with Spatial Control and Quantitative Feedback

Authors: Yao Zhang, Zhuchenyang Liu, Yu Xiao

Organizations: Aalto University

Abstract

Recent work on human-scene interaction (HSI) generation moves from single interactions to long-horizon interaction, where a language-model agent plans and executes a sequence of interactions in a 3D scene under one instruction. However, existing agents struggle in challenging everyday scenarios (e.g., picking up a toy and throwing it into a basket) because they lack spatial control over the motion and quantitative feedback on its execution, and they often fail to complete the interaction sequence. To address these gaps, we present Inhabitant, a training-free agent framework in which the agent specifies each motion through positions, paths and poses, measures how the executed interaction deviates from its intent, and revises the control specification accordingly. For an extensive evaluation in such scenarios, we build Inhabit, a benchmark of 75 household tasks in 31 furniture layouts of an apartment, in which every task chains one to three goals on movable objects and is carried out by the whole body, excluding fingers. On Inhabit, Inhabitant completes 98.7% of the tasks with Claude Opus 5.5, and 93.3% and 82.7% with Claude Sonnet 5.5 and Gemini 4 Argon, respectively. Ablations confirm that without spatial control the agent fails most tasks, and without quantitative feedback its motion becomes less precise. Code is available on the project page.

Figures & tables

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. World2HOSI: A Multi-Expert Harness for Long-Horizon Human-Object-Scene Interaction Generation

    Oct 20, 2026Jiushu Li, Haoran Wang, Yumeng Liu +4HOI Generation

  2. SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interaction

    Jun 25, 2026Sirui Chen, Shibo Zhao, Zhen Wu +3Robotic ControlHumanoid Robot Locomotion