Recent work on human-scene interaction (HSI) generation moves from single interactions to long-horizon interaction, where a language-model agent plans and executes a sequence of interactions in a 3D scene under one instruction. However, existing agents struggle in challenging everyday scenarios (e.g., picking up a toy and throwing it into a basket) because they lack spatial control over the motion and quantitative feedback on its execution, and they often fail to complete the interaction sequence. To address these gaps, we present Inhabitant, a training-free agent framework in which the agent specifies each motion through positions, paths and poses, measures how the executed interaction deviates from its intent, and revises the control specification accordingly. For an extensive evaluation in such scenarios, we build Inhabit, a benchmark of 75 household tasks in 31 furniture layouts of an apartment, in which every task chains one to three goals on movable objects and is carried out by the whole body, excluding fingers. On Inhabit, Inhabitant completes 98.7% of the tasks with Claude Opus 5.5, and 93.3% and 82.7% with Claude Sonnet 5.5 and Gemini 4 Argon, respectively. Ablations confirm that without spatial control the agent fails most tasks, and without quantitative feedback its motion becomes less precise. Code is available on the project page.
Figures & tables
Figure 1: One step of a household task: picking up a cube from the counter. (a) Existing agents control the motion through a text prompt or a skill and receive no feedback or only rendered frames; after the hand misses the cube, the agent can only reword the prompt, and the hand misses again. (b) Inhabitant measures how far the hand stopped from the cube (quantitative feedback), fixes where the right hand must be at a given frame (spatial control), and grasps the cube on the next try. The frames in (a) come from our agent restricted to text prompts.
Figure 2: Overview of Inhabitant. Left: the agent π controls a pretrained motion generator G by writing a control specification c (spatial control) and interaction events e . The scene simulator E executes the motion, and the agent queries measurements M(q,τ) of the candidate segment τ (quantitative feedback) before it tries again, commits or rolls back (Sec. 3.2 ). Right: one episode with Claude Opus 5.5, 13 committed segments out of 37 tries. Solid bodies: ends of committed segments, numbered in order; hollow bodies: discarded candidates; insets: three of the interactions; check mark: the success rule is satisfied.
Qualitative feedback o(τ) , after every try
Check
whether the success rule G is satisfied by the sequence extended by the candidate segment
Frames
eight frames of the candidate segment, each from two third-person cameras: one follows the body, one views the body and the target of the motion from the side
Quantitative feedback M(q,τ) , on a query
Body
(i) position of a body part; (ii) distance from a body part to an object, a surface or a handle; (iii) contacts of the body; (iv) pose that the body executed at a frame
Scene
(v) position of an object; (vi) distance between two objects or between an object and a region; (vii) extent of an object or a piece of furniture
Table 1: The four kinds of feedback that the agent receives on a candidate segment.
Figure 3: Overview of Inhabit . (a) Half-physics simulation: the hollow body shows the generated motion before execution, and the solid body shows the executed motion after scene interaction. (b) The five goal types, illustrated on segments from an Inhabitant episode. (c) Example tasks from each tier in the apartment, showing the body path, each object’s start position and its target. (d) Success rule for the high-tier task in (c), with one condition per goal. (e) The full set of 75 tasks: each circle represents one task, with one sector per goal colored as in (b), alongside three of the 31 furniture layouts.
Success rate (%) ↑
Motion quality ↓
Cost ↓
Method
Language model
All
low
mid
high
posture
Sub-goal
Collisions
Disturbed
Skate (%)
Tries
Calls
Tokens
InteractAgent † [ 60 ]
Claude Opus 5.5
26.7
53.3
10.0
0.0
20.0
34.5
105.0
2.4
0.9
52.8
133
123k
HumanCLAW † [ 37 ]
Claude Opus 5.5
20.0
33.3
15.0
6.7
10.0
26.8
42.1
0.8
3.6
11.9
12
91k
GEMS † [ 1 ]
Claude Opus 5.5
6.7
13.3
5.0
0.0
0.0
13.5
11.9
1.0
5.3
3.8
7
98k
Inhabitant
GLM-5.3-Flash
44.0
80.0
10.0
0.0
70.0
53.2
36.8
1.7
5.9
38.2
112
221k
gpt-6-luna
30.7
40.0
25.0
6.7
50.0
41.4
45.3
1.5
5.4
46.4
122
143k
Table 2: Results on the 75 tasks of Inhabit. † : baseline adapted to Inhabit. Bold: best success rate; for motion quality and cost, best among the runs of Inhabitant only, since the baselines stop early and return shorter sequences.
Figure 4: Inhabitant and the three baselines on one task with the same language model; chips show which of our two interfaces each agent has and what it has in place of a missing one. Red dots: where the throws of InteractAgent † land; yellow circle: the 10 cm radius of the grasp rule; inset: a missed grasp of GEMS † . More examples are in the video on the project page.
Control specification
Success
Collisions
Missed grasps
Tries
Calls
Atomic skills
30.0
8.4
68
12.6
32
Text prompts only
20.0
7.2
73
61.0
134
+ root waypoints
90.0
9.6
62
27.9
69
+ end-effector targets
95.0
7.7
45
17.4
51
+ full-body keyframes (Inhabitant)
95.0
7.8
40
14.8
47
Table 3: Granularity of spatial control on the ablation set with Claude Opus 5.5.
#
Check
Frames
Body
Scene
Success
Collisions
Missed grasps
Tries
Calls
1
✓
✓
85.0
14.6
58
22.7
54
2
✓
✓
✓
95.0
10.9
47
19.3
64
3
✓
✓
✓
95.0
12.4
61
22.0
59
4
✓
✓
✓
✓
95.0
7.8
40
14.8
47
5
✓
✓
✓
85.0
9.5
47
19.8
60
6
✓
70.0
11.1
53
21.6
54
Table 4: Granularity of feedback on the ablation set with Claude Opus 5.5 and all three types of spatial constraints. Check, Frames, Body and Scene are the four kinds of feedback of Tab. 1 .
Setting
Success
Collisions
Tries
Calls
Inhabitant
95.0
7.8
14.8
47
Single-Try
75.0
30.4
13.2
36
Open-Loop
0.0
10.9
4.2
2
Table 5: The execution loop on the ablation set with Claude Opus 5.5.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Call
Tool and arguments
Result
\textsctry(c,e)
generate_segment: text prompts, durations, constraints, seed, post-processing, guidance weights ( c ) and interaction events ( e )
success check and eight rendered frames of the try
\textscmeasure(q)
inspect: a list of queries
one value or time series per query
commit
commit
state description within 2 m of the body
\textscrollback(j)
retry_from: index of a committed segment
state description
–
get_state
full state description
Appendix
Table 6: The tools of Inhabitant and the calls of Eq. ( 1 ) that they implement.
Query
Kind
Returns
where
body, scene
position of a body part or an object, now, at a time of the try or every 0.1 s; for an object also the surface or hand it is on, for a handle whether it is open
distance
body, scene
distance between two entities (body parts, objects, surfaces, furniture) every 0.1 s, with the direction at the closest moment
contacts
body
contacts of the body with the scene during the try
pose
body
pose that the body executed at a time of the try, in the format of a full-body keyframe
extent
scene
corners and heights of an object, a surface or a piece of furniture
fk
–
joint positions of a pose written as a full-body keyframe, before it is used in a try (Sec. 3.3 )
Appendix
Table 7: The queries of a measurement and the kind of quantitative feedback of Tab. 1 that they give.
Generator
Root (m)
End effector (m)
Within 10 cm
Heading ( ∘ )
Time (s)
Kimodo, full control specification
0.035
0.005
100%
1.5
3.7
OmniControl
0.085
0.057
81%
39.9
95.4
Kimodo, text prompts only
1.886
0.708
0%
52.9
3.6
Appendix
Table 8: Generators on 120 control specifications of the agent: mean error of the root at its waypoints (214 targets) and of the constrained joints of the end-effector targets (133 targets), share of end-effector targets reached within 10 cm, mean heading error at the waypoints that set a heading, and generation time per segment on a V100. OmniControl has no heading input.
Tier
Tasks
with throw
by foot
EB original
low (1 goal)
30
5
5
3
mid (2 goals)
20
3
4
3
high (3 goals)
15
0
3
2
posture
10
–
–
0
total
75
8
12
8
Appendix
Table 9: Composition of the tasks: number of tasks that contain a throw goal or a goal of moving an object with a foot, and original EB-Habitat episodes. All other goals are place goals. The 10 posture tasks are sitting (4), lying (2) and kneeling (4). Goal counts of EB originals follow EB’s own goal.
Condition
Holds if
on
the object rests inside the region of the target surface and is not held
in
the centre of the object is inside the target box or basket, wherever that container is now, and the object is not held
inside
the object is inside the fridge and is not held
open
the fridge door is opened by more than 1.07 rad, or the drawer is pulled out by more than 0.35 m
closed
the joint of the door or drawer is within 0.15 of its closed position
throw
the object ends on or in the target, and the palm was at least 1 m (horizontally) from the target when the object left the hand
Appendix
Table 10: The conditions of the success rules. The target of a kick or a push can be the floor under a piece of furniture, where the object must lie less than 0.15 m above the floor.
Task
Instruction
Conditions
low_01
Could you tidy up the toy airplane? It goes in the cardboard box by the TV stand.
in
low_02
Put the lego in the cardboard box next to the black table.
in
low_03
Put the toy airplane in the cardboard box next to the black table.
in
low_04
The towel is on the floor. Put it back on the sofa.
on
low_05
There’s a towel lying on the floor - could you put it back on the sofa?
on
low_06
Pick up the remote control from the floor and put it on the TV stand.
on
Appendix
Table 11: The 30 low tasks of Inhabit: instruction and conditions of the success rule (Tab. 10 ). ∗ marks a task of the ablation set.
Task
Instruction
Conditions
mid_01
Could you tidy the toy airplane and the rubriks cube into the basket by the black table?
2 × in
mid_02
Could you tidy the lego and the toy airplane into the cardboard box by the sofa?
2 × in
mid_03
Tidy up: the rubriks cube goes in the basket next to the sofa; throw the toy airplane in from a metre or more.
in, throw
mid_04
The fridge is open. Put the banana inside and close the door.
inside, closed (ordered)
mid_05
The fridge is open. Put the peach inside and close the door.
inside, closed (ordered)
mid_06 ∗
Take both the coffee mug and the serving bowl on the black table to the sink in one go - one in each hand.
2 × on, together
Appendix
Table 12: The 20 mid tasks of Inhabit, as in Tab. 11 . “Ordered” means that the conditions must become true in the order of the instruction.
Task
Instruction
Conditions
high_01
Clear the sofa: all the things on it go on the top shelf of the tall shelf next to the black table.
3 × on
high_02
Clear the sofa: all the things on it go on the top shelf of the tall shelf next to the brown table.
3 × on
high_03 ∗
Take everything out of the fridge, put it on the black table, and close the fridge again.
2 × on, closed
high_04
Empty the fridge onto the black table, then shut the fridge door.
2 × on, closed
high_05
Put all the things on the black table into the cardboard box beside it.
3 × in
high_06
Clear everything off the black table into the cardboard box next to it.
3 × in
Appendix
Table 13: The 15 high tasks and the 10 posture tasks of Inhabit, as in Tab. 11 .
Failed
Budget
Stopped
Plan
Missed
Baseline
episodes
used up
by the agent
executed
grasps
InteractAgent †
55
46
9
–
83%
HumanCLAW †
60
0
60
–
75%
GEMS †
70
–
–
70
68%
Appendix
Table 14: How the failed episodes of the baselines of Tab. 2 end, and their missed grasps over all 75 episodes (Supp. B.3 ). GEMS † has one call for its plan, so its episodes end when the plan is executed.
Tokens per episode
Cost (USD)
Method
Language model
Input
Cached
Output
Episode
Solved task
InteractAgent †
Claude Opus 5.5
6.3M
98%
39k
2.56
9.62
HumanCLAW †
Claude Opus 5.5
1.5M
95%
16k
0.96
4.80
GEMS †
Claude Opus 5.5
1.3M
92%
32k
1.41
21.08
Inhabitant
GLM-5.3-Flash
15.4M
0%
106k
–
–
gpt-6-luna
10.5M
87%
51k
0.25
0.82
Appendix
Table 15: Cost in API fees of the runs of Tab. 2 .
Task
IA †
HC †
GEMS †
GLM
luna
Gemini
Sonnet
Opus
Task
IA †
HC †
GEMS †
GLM
luna
Gemini
Sonnet
Opus
low_01
✓
✓
⋅
✓
⋅
✓
✓
✓
mid_09
⋅
⋅
⋅
⋅
✓
✓
✓
✓
low_02
✓
⋅
⋅
✓
⋅
✓
✓
✓
mid_10
⋅
⋅
⋅
⋅
⋅
✓
✓
✓
low_03
⋅
⋅
⋅
✓
⋅
✓
✓
✓
mid_11
⋅
⋅
⋅
⋅
⋅
✓
✓
✓
low_04
✓
✓
✓
✓
✓
✓
✓
✓
mid_12
⋅
⋅
⋅
⋅
⋅
⋅
✓
✓
low_05
✓
✓
⋅
✓
✓
✓
✓
✓
mid_13
⋅
⋅
✓
⋅
⋅
✓
✓
✓
low_06
⋅
✓
⋅
✓
✓
✓
✓
✓
mid_14
⋅
⋅
⋅
⋅
⋅
✓
✓
✓
Appendix
Table 16: Per-task results of the runs of Tab. 2 : ✓ marks a success. IA † , HC † and GEMS † are the three baselines with Claude Opus 5.5, and the other columns are Inhabitant with the five language models. ∗ marks a task of the ablation set. The last row counts the solved tasks out of 75.
Opus 5.5
Sonnet 5.5
Gemini
GLM
luna
InteractAgent †
Segments with a spatial constraint
97%
96%
100%
90%
90%
–
with an end-effector target or a full-body keyframe
37%
26%
49%
17%
4%
–
Committed at the first try
46%
52%
48%
46%
36%
32%
Tries followed by a measurement
93%
99%
98%
80%
95%
–
Revisions that change only the seed
6%
7%
0%
1%
1%
–
Missed grasps
48%
66%
23%
73%
79%
83%
Appendix
Table 17: How the agent uses the spatial constraints and the feedback in the episodes of the main table, for Inhabitant with the five language models and for InteractAgent † , which has text prompts only and neither measurement queries nor a seed. Revisions are pairs of consecutive tries for the same segment. The rows on grasps take every try whose grasp failed and ask whether the grasp of the next try for the same segment succeeds, split by whether the agent changed an end-effector target or a full-body keyframe in between. The last rows show how the failed episodes end. –: does not apply; n/a: no such case.
Task
Skills
Text
+ root
+ end eff.
+ keyfr.
low_07
⋅
⋅
✓
✓
✓
low_11
⋅
⋅
✓
✓
✓
low_12
⋅
⋅
✓
✓
✓
low_16
✓
✓
✓
✓
✓
low_18
⋅
⋅
✓
✓
✓
low_21
⋅
⋅
✓
⋅
✓
Appendix
Table 18: Per-task results of the settings of Tab. 3 on the ablation set, in the order of its rows: atomic skills, text prompts only, and the three types of spatial constraints added one at a time. ✓ marks a success.
Task
1
2
3
4
5
6
low_07
⋅
✓
✓
✓
✓
⋅
low_11
✓
✓
✓
✓
✓
✓
low_12
✓
✓
✓
✓
✓
✓
low_16
✓
✓
✓
✓
✓
✓
low_18
✓
✓
✓
✓
✓
✓
low_21
✓
✓
✓
✓
✓
⋅
Appendix
Table 19: Per-task results of the six settings of Tab. 4 on the ablation set, numbered as its rows. ✓ marks a success.
Success check
All
low
mid
high
posture
Sub-goal
Collisions
Disturbed
Calls
Tokens
with
98.7
100.0
100.0
100.0
90.0
99.6
9.3
1.0
51
99k
without
86.7
96.7
85.0
80.0
70.0
92.0
8.2
0.7
58
128k
Appendix
Table 20: Inhabitant with Claude Opus 5.5 and all measurements on the 75 tasks of Inhabit, with and without the success check. Success rates and sub-goals in %, the other columns are means per episode as in Tab. 2 . Without the check the agent ends the episode itself, and the episode is judged on its final state.
Control specification
Full loop
Single-Try
Open-Loop
Full (Inhabitant)
19 (7.8)
15 (30.4)
0 (10.9)
Text prompts only
4 (7.2)
8 (178.3)
0 (23.2)
HumanCLAW skills
6 (8.4)
6 (23.9)
1 (20.4)
Full, no measurements
17 (14.6)
Appendix
Table 21: All ten settings on the ablation set with Claude Opus 5.5: tasks solved out of 20 (mean collisions per task).
Figure 8: Success against collisions per committed segment for the ten settings; marker area is the mean number of tool calls per task.
Windows (frames covered)
171 (73%)
Tracked without failure
161 (94%)
Global MPJPE
53.7 mm
Root-relative MPJPE
40.5 mm
Acceleration error
6.0 mm/frame 2
Global MPJPE, walking
52.8 mm
Global MPJPE, low postures
110.8 mm
Appendix
Table 22: PHC+ tracking of the interaction sequences of Claude Opus 5.5 on all 75 tasks, without furniture and objects. Low postures are frames with the head below 1.2 m: bending, squatting, kneeling and lying.
Long-horizon human-scene interaction requires coordinated spatial reasoning, physical feasibility, and the ability to recover from compounding execution errors. Text-driven motion models are largely scene-agnostic and lack closed-loop correction, while geometry-conditioned approaches depend on paired scene-motion training or explicit 3D preprocessing that limits generalization to novel layouts and open-vocabulary goals. We present InteractAgent, an agentic multimodal LLM system for long-horizon human-scene interaction that plans, executes, observes, and reflects in a closed loop, using a single MLLM as both planner and evaluator and a frozen MotionLLM as the executor. InteractAgent integrates Online Observation for step-wise recovery decisions (continue/redo/replan) and Memory-Augmented Reflection for multi-round plan refinement with adaptive stopping. On UniHSI, InteractAgent improves task success and reduces penetration compared with training-free baselines. Code: https://github.com/AIGeeksGroup/InteractAgent.
Long-horizon human-object-scene interaction (HOSI) is central to embodied AI, animation, and game development. Existing human-object interaction (HOI) and human-scene interaction (HSI) models excel in their respective domains, but none alone covers tasks requiring multiple interaction skills. Training unified HOSI models is hindered by scarce and costly data. We introduce World2HOSI, a multi-expert orchestration harness that generates continuous HOSI data from 3D Gaussian Splatting (3DGS) scenes and natural language instructions. Our key insight is that long-horizon HOSI hinges not on a stronger single model but on orchestration. Given a shared, structured scene state and an orchestration harness grounded in it, existing experts can jointly accomplish tasks beyond the reach of any of them alone. World2HOSI realizes this insight as a closed loop around a persistent scene state built from decoupled background and object assets. A harness plans action stages over this state, invokes experts via their native conditioning interfaces, and writes each action's effects back, so that every invocation is grounded in the evolving world. To reduce physical artifacts, bounded residual optimization and contact-driven physics simulation further ensure the physical plausibility of human motion and object manipulation, respectively, while preserving interaction intent. Experiments and a perceptual user study show that World2HOSI achieves the best task completion and interaction quality on the evaluated HOSI tasks, while generating more diverse data.
Jiushu Li, Haoran Wang, Yumeng Liu +4
Macau University of Science and Technology · University of Science and Technology of China · Massachusetts Institute of Technology +2
Current humanoid reinforcement-learning policies excel at free-space motions but struggle with contact-rich tasks, as pure kinematic tracking cannot resolve the physical ambiguities of interacting with objects and uneven terrain. To address this, we introduce SceneBot, a unified motion-tracking framework capable of handling freespace locomotion, terrain traversal, and whole-body manipulation. SceneBot conditions a single policy on both reference motions and per-link contact labels, explicitly defining expected environmental interactions. To overcome the lack of annotated interaction data, we propose a hindsight scene reconstruction approach that infers scene-interaction graphs from retargeted human motion. Trained on 7.5 hours of this reconstructed, contact-rich data, SceneBot successfully generalizes to unseen motions and environments. Our results demonstrate that SceneBot is the first general framework to seamlessly unify free-space and contact-rich behaviors executing complex, long-horizon tasks like carrying a box upstairs and establishing contact conditioning as a powerful interface for humanoid control. All code and data will be open-sourced. More demos and information are available at: https://ericcsr.github.io/scenebot/
Sirui Chen, Shibo Zhao, Zhen Wu +3
Stanford, Amazon FAR United States · Amazon FAR United States · CMU, Amazon FAR United States